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Abstract 


1. 


A  feature  of  current  sequential  quadratic  programming  (SQP)  methods 
to  solve  nonlinear  constrained  optimization  problems  is  the  necessity  at 
each  iteration  to  solve  a  quadratic  program  (QP).  We  show  that  if  the  QP 
subproblem  is  convex  and  an  active-set  method  is  used  to  solve  it,  then 
there  exist  iterates  other  than  the  minimizer  that  may  be  used  to  define  a 
suitable  search  direction.  None  of  the  usual  properties  of  an  SQP  method 
are  compromised  by  the  new  definition  of  the  search  direction. 

We  derive  some  new  properties  for  an  SQP  method  that  uses  a  particular 
augmented  Lagrangian  merit  function.  Specifically  we  show,  under  suitable 
additional  assumptions,  that  the  rate  of  convergence  is  superlinear.  We  also 
show  that  the  penalty  parameter  used  in  the  merit  function  is  bounded. 


Introduction 


The  problem  of  interest  is  the  following: 


NP 
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where  F  :  Si"  — »  3?  and  c  :  SJn  — *  3?m.  More  specifically,  we  are  interested  in  computing  x* , 
a  first-order  KKT  point  of  NP.  Such  points  are  feasible  and  satisfy  the  following  conditions: 

VF(x*)  =  Vc(x*)TA*,  A*c,(z*)  =  0  j  =  1, . . . ,  m 


for  some  nonnegative  multiplier  vector  A*  £  $Jm.  Since  we  shall  not  assume  second  deriva¬ 
tives  are  known,  this  is  the  best  that  can  be  achieved,  and  so  whenever  the  term  “solution 
point”  is  used  in  the  following  sections,  what  will  be  meant  is  a  first-order  KKT  point  of 
NP. 

We  study  the  convergence  properties  of  a  sequential  quadratic  programming  (SQP) 
algorithm  in  which  the  quadratic  programming  subproblems  generated  are  convex.  The 
unique  feature  of  the  algorithm  studied  is  that  the  search  direction  in  a  given  iteration  is 
computed  as  an  “incomplete  solution”  of  the  quadratic  subproblem.  A  precise  definition  of 
the  term  “incomplete  solution”  will  be  given  in  Section  2. 

Typically  SQP  algorithms  generate  a  sequence  of  points  {z^}  converging  to  a  solution, 
by  solving  at  each  point,  x *,  a  quadratic  program  (QP)  of  the  form 


minimize  V  F(xk)Tp  +  \pTHkP 
s.t  c(ifc)  +  Vc(zjt)p  >  0 


for  some  positive  definite  matrix  H Let  p*  (referred  to  as  the  search  direction)  denote  the 
solution  to  QP.  The  next  point  in  the  sequence  is  defined  to  be  the  result  of  a  linesearch 
from  zjt  along  pk,  in  such  a  way  that  the  value  of  a  certain  merit  function  is  decreased. 

A  number  of  different  merit  functions  have  been  proposed.  The  one  studied  in  this 
paper  is  that  analysed  in  [GMSW86b]  and  used  in  the  algorithm  NPSQP.  It  is  similar  to 
merit  functions  proposed  by  Wright  [Wri76]  and  Schittkowski  [Sch81]. 

Although  our  primary  interest  is  this  specific  merit  function,  we  also  show  (Section  5) 
how  the  ideas  discussed  can  be  extended  to  the  use  of  other  merit  functions. 

SQP  algorithms  in  general  and  NPSQP  in  particular  are  viewed  by  many  as  the  best 
approach  to  the  solution  of  NP  when  n  is  small  (  <  200  ).  As  the  size  of  the  problem  grows, 
usually  so  does  the  relative  importance  of  the  effort  to  solve  QP  when  compared  to  the  total 
effort.  Indeed  for  many  large  problems  the  effort  to  solve  QP  dominates  the  total  effort. 

Because  the  unique  minimizer  of  QP  is  used  to  define  the  search  direction,  it  is  not 
necessary  in  any  theoretical  discussion  of  an  SQP  algorithm  to  define  how  the  QP  sub¬ 
problem  is  solved.  In  practice  by  far  the  commonest  techniques  used  to  solve  QPs  are 
active-set  methods.  The  implementation  of  NPSQP  called  NPSOL  uses  such  a  method. 
For  a  comprehensive  survey  of  active-set  methods  see  [GMW81],  [Fle87]  and  [GMSW88]. 
When  an  active-set  method  is  used  the  potential  number  of  iterations  to  solve  a  QP  grows 
exponentially  with  n.  In  practice  the  number  of  iterations  grows  much  more  slowly  than 
exponential  (if  this  was  not  the  case  active-set  methods  would  be  hopelessly  inefficient). 
Nonetheless,  the  number  of  iterations  required  to  solve  a  large  QP  is  usually  large.  In  any 
implementation  of  an  SQP  method  it  is  necessary  to  limit  the  number  of  iterations  allowed 
to  solve  a  given  QP  subproblem.  If  the  QP  solution  process  is  terminated  prematurely  the 
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SQP  algorithm  may  break  down.  It  is  in  part  for  this  reason  that  the  development  of  SQP 
methods  for  large-scale  problems  has  been  inhibited.  Even  for  small  problems  there  are 
occasions  when  the  number  of  QP  iterations  are  excessive.  Since  the  definition  of  “small” 
continues  to  increase  as  computers  become  more  powerful  we  can  expect  the  cost  of  solving 
the  subproblems  to  grow  in  importance. 

Typically  the  number  of  iterations  to  solve  the  QP  subproblem  is  large  in  the  initial  it¬ 
erations  and  falls  exponentially.  Near  the  solution  only  a  single  iteration  is  usually  required. 
A  natural  question  is  whether  the  effort  to  solve  the  QP  subproblem  is  warranted  far  from 
the  solution  when  the  information  used  to  construct  the  subproblem  may  be  unreliable.  We 
present  an  alternative  definition  of  the  search  direction  that  is  not  based  on  the  minimizer 
of  QP. 

Our  goals  can  be  summarized  as  being: 

•  The  derivation  of  a  global  convergence  proof  for  the  algorithm. 

•  A  proof  of  superlinear  convergence  under  additional  assumptions. 

•  A  proof  that  the  penalty  parameter  used  in  the  merit  function  is  bounded. 


Incomplete  solutions  for  QP  subproblems 

The  great  majority  of  SQP  algorithms  in  the  literature  define  the  search  direction  from  a 
minimizer  of  the  QP  subproblem,  although  there  have  been  some  proposals  to  terminate 
the  solution  process  for  the  QP  subproblems  early.  An  approach  solving  QP  subproblems 
inexactly  is  described  in  Dembo  and  Tulowitzki  [DT85],  where  for  a  generic  SQP  algorithm 
an  early  termination  rule  is  given  in  terms  of  the  norm  of  the  reduced  gradient  for  the 
subproblem.  This  rule  gives  a  search  direction  pk  satisfying  the  condition 


lip*  -  pI\\  =  °(IMI)> 


where  p*k  denotes  the  minimizer  for  the  fcth  QP  subproblem. 

We  follow  a  different  approach,  presenting  an  early  termination  rule  that  is  constructive 
in  nature,  and  that  has  a  guaranteed  bound  on  the  effort  necessary  to  satisfy  it.  If  the 
solution  process  is  terminated  early,  the  search  direction  for  the  outer  iteration  (the  step  in 
the  original  variables)  is  defined  as  the  “total”  step  taken  in  the  QP  subproblem  up  to  that 
point.  The  characteristics  of  the  point  at  which  the  termination  takes  place  clearly  depend 
on  the  specific  strategy  used  to  solve  the  QP  subproblem.  In  the  course  of  solving  a  QP  an 
active-set  method  generates  iterates  that  are  stationary  points.  We  show  that  such  points 
may  be  used  to  construct  a  suitable  search  direction. 

Terminating  the  QP  algorithm  prior  to  obtaining  a  solution  impacts  the  SQP  algorithm 
in  a  number  of  critical  ways.  Not  only  is  the  search  direction  different,  but  also  the  QP 
multipliers  will  not  be  available.  The  merit  function  of  interest  requires  the  definition  of  a 
search  direction  in  the  space  of  the  multipliers.  In  NPSQP  this  direction  is  defined  using 
the  QP  multipliers.  The  proof  of  convergence  for  NPSQP  makes  use  of  the  fact  that  the 
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QP  multipliers  are  positive.  The  consequences  of  terminating  the  QP  solution  process  early 
are  therefore  far  reaching. 

The  remainder  of  this  paper  is  organized  as  follows.  Section  2  describes  the  form  of  the 
general  algorithm,  and  presents  the  conditions  on  the  search  direction  and  the  multiplier 
estimate.  Section  3  studies  the  convergence  properties  of  the  algorithm;  it  is  shown  that 
such  an  algorithm  is  globally  convergent.  In  Section  4  we  show  that  the  algorithm  converges 
superlinearly.  We  also  show  that  the  penalty  parameter  used  in  the  merit  function  is 
bounded.  Section  5  establishes  the  convergence  of  algorithms  that  use  merit  functions 
belonging  to  a  certain  class  (that  includes  the  merit  function,  for  example).  Finally, 
Section  6  presents  numerical  results  obtained  from  an  implementation  of  the  first  algorithm. 

2.  Description  of  the  algorithm 

This  section  introduces  the  proposed  algorithm.  The  algorithm  obtains  the  search  direction 
from  an  incomplete  solution  of  QP.  The  iterates  are  determined  by  conducting  a  linc  earch 
on  the  following  merit  function: 

(2.1) 

where  s  >  0  are  slack  variables,  and  the  scalar  p  is  known  as  the  penalty  parameter. 

The  search  is  performed  on  an  expanded  space,  including  the  Lagrange  multiplier  es¬ 
timates  A,  and  the  slack  variables  s.  The  symbols  p,  £  and  q  will  be  used  to  denote  the 
components  of  the  search  direction  on  the  corresponding  subspaces.  In  this  case,  the  value 
of  the  merit  function  as  a  function  of  the  steplength  will  be  denoted  by 

<t>(a)  =  LA{x  +  ap,\  +  a£,s  +  aq,p). 

The  derivative  of  (f>  with  respect  to  a  is  denoted  by  <f>'. 

The  following  conventions  will  be  used  in  the  rest  of  the  paper: 

gk  =  V.F(xfc),  Ak  =  Vc(xfc),  ck  =  c(x  k), 

although  the  last  two  symbols,  Ak  and  ck,  will  also  be  used  with  the  same  meaning  but 
restricted  to  the  set  of  active  constraints  at  the  given  point.  The  term  active  constraint  will 
be  used  to  designate  a  constraint  that  is  satisfied  exactly  at  the  current  point  (c,(x)  =  0 
in  NP,  or  ajp  =  -c<  in  QP),  and  the  set  of  all  constraints  active  at  a  given  point  will  be 
referred  to  as  the  active  set  at  the  point. 

The  objective  function  for  the  QP  subproblem  will  be  denoted  by  ^jt(p), 

Mp)  =  9kP  +  \pTHkP- 

Sometimes,  xp  will  denote  the  function  of  one  variable  V’fc(a)  =  0*(p  +  ad).  Finally,  sym¬ 
bols  of  the  form  /3a(,c  indicate  fixed  scalars  related  to  properties  of  the  problem,  or  the 
implementation  of  the  algorithm,  where  “chc”  identifies  the  specific  scalar  represented. 


(2.2) 


LA(x,\,s,p)  =  F(x)  -  A T(e(x)  -  s)  +  ^p(c(x)  -  s)T(c(x)  -  s)  , 
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The  algorithm 

The  following  is  an  outline  of  the  algorithm,  the  details  are  given  in  subsequent  sections. 

Let  {Hk}  denote  a  sequence  of  symmetric  matrices  whose  smallest  eigenvalue  is  positive 
and  uniformly  bounded  away  from  zero  and  whose  largest  eigenvalue  is  uniformly  bounded. 
The  algorithm  proceeds  through  the  following  steps: 

(i)  Set  p_ i  =  0.  Choose  xo,  Ao  and  po  >  0.  Set  k  =  0. 

(ii)  At  each  point  form  the  QP  subproblem 


minimize  gJp+\pTHkP 

pe»n  1 

subject  to  AkP  >  - Ck , 


and  from  an  incomplete  solution  determine  pk  satisfying  certain  conditions.  Compute 
a  vector  of  multipliers  pk  satisfying  a  second  set  of  conditions.  The  precise  conditions 
that  pk  and  pk  need  to  satisfy  are  given  later  in  this  section.  If  pk  =  0,  set  A*  =  Pk 
and  terminate.  Otherwise,  compute  &  =  Pk  -  A*. 

(iii)  Compute  sk  from 


(Sk)i 


max(0,{cfc)«) 

max(0,(cfc),  - 

v  Pk- x ' 


if  pfc-i  =  0, 
otherwise. 


Find  pk  such  that  <£'fc(0,p/t)  <  -\plHkPk- 
Compute  qk  from 


Pk  =  AkPk  +  Ck  -  Sk- 


(2.3) 


(iv)  Compute  the  steplength  ct*.  The  termination  conditions  for  the  linesearch  are  as 
follows: 

If 

4>(l)  -  <t>(0)  <  (2.4) 

set  a*  =  1.  Otherwise,  find  an  Qk  €  (0, 1)  such  that 

<fr(otk)  ~  4>(0)  <  <XQk<f>'(0)  (2.5a) 

4>'(ok)  >  #'(0),  (2.5b) 

where  0  <  cr  <  p  <  5. 

(v)  Update  Xk  and  At  using 


'  Xk+ 1  ^ 

'  xk  N 

(  Pk  \ 

Ajt+i 

= 

At 

+  Qk 

& 

^  ) 

V  / 

\  / 

Set  k  k  +  1  and  repeat  steps  (ii)  to  (v)  until  convergence  is  reached. 
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The  definition  of  the  search  direction 

In  each  iteration  the  search  direction  is  constructed  from  the  information  obtained  when 
solving  the  QP  subproblem  by  an  active-set  method.  Information  available  at  the  point  the 
algorithm  is  terminated  is  used  to  construct  p*  in  the  way  indicated  by  the  following  steps. 
We  have  omitted  the  subscript  k  corresponding  to  the  iteration  number. 

(i)  An  initial  feasible  point  p°  for  the  QP  subproblem  is  obtained. 

If  the  minimizer  for  the  QP  is  used  to  determine  the  search  direction,  then,  given 
the  uniqueness  of  pjt,  the  choice  of  p°  is  irrelevant.  When  an  incomplete  solution  of 
the  QP  subproblem  is  used  to  define  the  search  direction,  the  choice  of  p°  becomes 
critical.  If  we  determine  the  search  direction  from  a  stationary  point  that  is  not  a 
minimizer,  the  sequence  of  stationary  points  that  we  compute  depends  directly  on  the 
value  of  p°.  We  wish  to  define  the  initial  point  in  such  a  manner  that  all  stationary 
points  are  satisfactory  points  at  which  to  terminate  the  solution  process.  We  require 
||p°||  to  be  small  whenever  the  points  Xk  are  close  to  i,  a  stationary  point  of  NP. 

Define  the  vectors  s  and  r  to  have  components 

s,  =  max(0,c,  -  p;), 

Cj  —  s,-  if  |c,-  —  s,j  <  |cj  —  s,j, 
c,  -  Si  otherwise; 

where  p  denotes  a  multiplier  estimate  such  that  the  following  property  holds: 

|jifc  -  i||  -»  0  =»  ||cfc  -  sk\\  — •  0. 

From  this  definition,  r  has  the  following  property: 

lie'll  <||r||<||c-s||. 

The  following  condition  on  p°  ensure  our  objective: 

•  For  some  constant  (3pca  >  0, 

llp°ll  <  /WM|.  (2-7) 

(ii)  A  sequence  of  feasible  descent  steps  are  taken  and  the  QP  algorithm  is  terminated  at 
a  stationary  point  p  for  the  QP  subproblem. 

(iii)  If  the  stationary  point  is  a  second-order  KKT  point  of  the  QP  subproblem,  the  search 
direction  is  computed  as  p  =  p. 

(iv)  If  the  stationary  point  is  not  a  second-order  KKT  point,  the  multiplier  vector  at  the 
stationary  point  must  have  some  components  that  are  negative.  In  this  case,  we  need 
to  compute  a  vector  d  and  a  scalar  a  with  the  following  properties: 


(2.6) 
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•  d  is  feasible  with  respect  to  the  active  constraints,  Ad  >  0,  and  its  norm  is 
bounded  above  and  below,  that  is,  for  some  constants  0und  >  filnd  >  0  it  holds 
that  f3und  >  ||<f||  >  (3[nd.  It  is  assumed  without  loss  of  generality  that  0lnd  <  1, 
in  order  to  simplify  the  arguments  in  the  proofs. 

•  The  rate  of  descent  along  d  is  sufficiently  large.  If  0(C)  =  0(p+C<O,  it  is  required 
that 

0'(O)  =  (Hp  +  g)Td  <  -(3dac max;/z~  (2-8) 


for  some  constant  fid3c  >  0. 

•  The  step  a  is  taken  as  the  step  to  the  minimizer  of  0(C),  if  it  is  feasible.  We 
have 

a  =  min(ac,am,OfM), 


where  ac  is  the  step  to  the  nearest  inactive  constraint,  aM  >  0  is  a  specified 
bound  on  the  largest  acceptable  step,  and 


0'(O) 
am  "  dTHd' 


(2.9) 


It  is  a  desirable  property  to  avoid  having  search  directions  with  very  small  norms,  un¬ 
less  the  corresponding  point  is  close  to  a  solution.  The  following  definition  is  sufficient 
to  ensure  this  property: 

j  p  + ad  if  ||p||  <  l3stp\\p  +  ad||,  (  , 

**  |  p  otherwise,  ' 

for  some  constant  f3,ip  >  0.  It  will  be  assumed  that  (3aip  is  chosen  so  that  (3,ip  >  1. 


The  multiplier  estimates 

Step  (ii)  of  the  algorithm  requires  not  only  a  search  direction  pk,  but  also  an  estimate  pk 
of  the  Lagrange  multipliers  at  the  current  point.  The  QP  algorithm  may  terminate  at  a 
stationary  point,  so  a  natural  choice  would  be  to  use  the  multipliers  of  the  stationary  point 
as  the  estimate,  but  in  general  these  may  not  be  the  best  possible  choice,  as  they  may  be 
negative,  or  the  active  set  associated  with  the  search  direction  may  not  in  some  cases  be 
the  same  as  the  one  for  which  the  multiplier  was  obtained.  The  following  conditions  on  //* 
are  sufficient  to  ensure  that  the  algorithm  has  the  desired  convergence  properties. 

MCI.  The  estimates  are  uniformly  bounded  in  norm. 

MC2.  -  A* ||  =  0(||p<;|j),  where  A*  denotes  the  multiplier  vector  associated  with  the 
solution  point  closest  to  x*,. 

MC3.  The  complementarity  condition  pJfAkPk  +  c^)  =  0  is  satisfied  at  all  iterations. 

Condition  MCI  and  the  form  in  which  the  multiplier  estimates  are  updated  imply  {A^} 
are  uniformly  bounded.  This  result  is  given  as  Lemma  4.2  in  [GMSW8ot>j. 
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Lemma  2.1.  For  all  k  >  0, 

M  <  max  ||/ij||, 

-1  1 

and  hence  |jA*||  is  bounded  for  all  k  (by  convention  p-\  =  X0). 

Second-order  information 

The  condition  imposed  on  {Hk}  is  that  the  matrices  in  the  sequence  are  positive  definite  and 
bounded,  with  bounded  condition  number.  This  assumption  is  identical  to  the  one  made 
for  NPSQP.  In  practice,  such  a  sequence  may  be  generated  (see  [GMSW86a])  by  updating 
a  quasi-Newton  approximation  to  the  Hessian  of  the  Lagrangian  function  or  the  Hessian  of 
the  augmented  Lagrangian  function  in  each  iteration  together  with  certain  safeguards. 

From  this  condition,  some  quantities  will  be  uniformly  bounded  in  the  algorithm.  The 
notation  introduced  below  is  used  throughout  for  these  bounds. 

(3lvH  is  an  upper  bound  for  the  largest  eigenvalue  of  II :  pTHp  <  I8ivh\\p\\2- 
PsvH  is  a  positive  lower  bound  for  the  smallest  eigenvalue  of  //:  pTIIp  >  /?j„//||p||2- 

3.  Global  convergence  results 

The  results  in  this  section  establish  global  convergence  properties  for  the  SQP  algorithm 
under  study. 

Assumptions  and  bounds 

We  make  the  following  assumptions: 

Al.  x k  lies  in  a  closed,  bounded  region  Q  C  ft",  for  all  k. 

A2.  F,  c,  and  their  first  and  second  derivatives  are  continuous  and  uniformly  bounded  in 
norm  on  ft. 

A3.  The  Jacobian  corresponding  to  the  active  constraints  at  any  limit  point  of  the  sequence 
generated  by  the  algorithm  has  full  rank. 

A4.  A  feasible  point  exists  to  all  the  QP  subproblems. 

A5.  Strict  complementarity  holds  at  all  stationary  points  of  NP  in  ft. 

A6.  The  reduced  Hessian  of  the  Lagrangian  function  is  nonsingular  at  all  first-order  KKT 
points  of  NP. 

From  these  assumptions,  several  quantities  are  uniformly  bounded  in  the  algorithm. 
The  first  two  bounds  follow  from  assumption  A2;  the  third  follows  from  A3. 

0nmc  is  a  bound  for  the  norm  of  the  constraint  vector:  ||cfc||  <  /?nmc. 
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Pnmg  ifi  a  bound  for  the  norm  of  the  gradient:  ||p*||  <  /?„ma. 

0nmu  is  an  upper  bound  for  the  norm  of  the  multipliers  corresponding  to  a  minimizer  of 
the  QP  subproblem:  ||pfc||  <  /?nmu- 

Properties  of  the  search  direction 

The  first  set  of  results  explores  the  relationship  of  stationary  points  of  the  QP  subproblems 
and  stationary  points  of  NP.  The  significance  of  this  relationship  is  due  to  the  fact  that 
the  search  direction  is  obtained  from  information  available  at  a  stationary  point  of  the  QP 
subproblem.  The  results  shown  below  are  similar  in  spirit  to  those  in  Robinson  [Rob74]. 
They  will  be  used  to  show  that  the  value  of  ||pfe||  is  “small”  if  and  only  if  we  are  close  to 
a  solution  point,  with  corresponding  implications  regarding  the  identification  of  the  correct 
active  set. 

Lemma  3.1.  For  any  x  £  Cl,  let  p  be  a  stationary  point  of  the  QP  subproblem  at  x.  For 
all  e  >  0  there  exist  a  constant  6  >  0  and  a  point  x  such  that 

IIpII  <  6  =>  \\x  -  i||  <  c, 

where  x  is  a  stationary  point  for  the  NP,  with  the  same  set  of  active  constraints  as  p,  if  the 
Jacobian  of  the  active  constraints  at  x  is  nonsingular. 

Proof.  Assume  that  the  result  does  not  hold;  then  there  exist  sequences  {pk}<j?=\  ,  and 
{xfc}fcLi  i  such  ^at  pk  is  a  stationary  point  of  the  QP  subproblem  at  ijt  satisfying  ||pfc||  — ►  0, 
and  ||x*  -  i||  >  €  for  some  e  >  0. 

A  convergent  subsequence  can  be  extracted  from  {x^},  using  the  compactness  of  Cl. 
Select  now  a  sub-subsequence  having  fixed  active  set  that  is  a  subset  of  the  active  set  at 
the  limit  point  x.  Such  a  subseqence  must  exist  since  there  only  a  finite  number  of  choices 
of  active  sets. 

If  we  take  limits  along  this  subsequence  of 

AkPk  +  Ck  >  0 

it  follows  from  ||p*||  — *  0  and  assumption  A2  that  x  must  be  feasible. 

If  the  Jacobian  of  the  active  constraints  is  nonsingular  at  x ,  it  follows  from 

HkPk  +  9k  =  A* pk 

that  there  exists  a  subsequence  along  which  {p*}  converges,  pjt  — ►  p.  Taking  limits  along 
this  subsequence, 

9  =  ATp. 

This  result  implies  that  x  is  a  stationary  point  of  NP,  contradicting  the  assumption. 

To  show  that  the  set  of  active  constraints  should  be  the  same  for  p  and  i,  assume  that 
sequences  as  described  above  exist,  but  that  the  set  of  active  constraints  at  each  p*,  is  not 
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the  same  as  the  set  of  active  constraints  at  x.  As  ||pjfe||  — ►  0,  the  set  of  active  constraints 
at  each  pk  must  be  a  subset  of  the  active  constraints  at  i;  but  if  it  is  a  proper  subset,  then 
there  must  exist  an  index  i,  active  at  x ,  such  that  {pk)i  =  0  for  large  enough  k,  and  this 
will  imply  p. ,  =  0,  violating  the  strict  complementarity  assumption.  | 

The  assumptions  on  the  form  of  the  problem  guarantee  that  large  enough  steps  can 
be  taken  from  stationary  points  in  the  QP  subproblems  when  the  points  considered  are 
not  close  to  solutions  for  the  problem.  The  algorithm  makes  use  of  this  property  to  move 
away  from  stationary  points  for  NP.  The  next  result  establishes  the  existence  of  some  of  the 
necessary  bounds. 

Lemma  3.2.  There  exist  positive  values  fi3pc  and  03pm,  such  that  for  all  stationary  points 
i  that  have  inactive  constraints 

min  c,  >  /3spc, 

i:c,  >0 

and  for  those  stationary  points  having  some  negative  multiplier  element, 

max/ij  >  0spm- 


Proof.  Assume  that  there  exists  a  sequence  {i^}  of  stationary  points  of  NP  in  SI  such  that 


min  cki 
>o 


0. 


From  the  compactness  of  SI,  a  convergent  subsequence  can  be  extracted  having  fixed 
active  set,  and  such  that  the  minimum  is  always  achieved  for  the  same  constraint  (or  set  of 
constraints).  Let  x*  denote  the  limit  point.  It  follows  from  Lemma  3.1  and  assumption  A3 
that  x*  is  a  stationary  point  of  NP.  At  this  point  assumption  A5  will  be  violated,  as  the 
corresponding  constraints  are  active,  but  have  zero  multipliers. 

If  the  sequence  is  such  that 

maxpjT  — >  0, 

using  the  same  construction  assumption  A5  will  again  be  violated  at  x* ,  as  at  least  one  of 
the  multipliers  corresponding  to  an  active  constraint  will  be  zero.  I 

We  now  proceed  to  establish  that  if  the  norm  of  the  search  direction  in  any  given  iteration 
||pfc||  is  small  enough,  then  the  correct  active  set  must  have  been  identified.  If  the  norm  of 
the  stationary  point  where  the  search  direction  is  computed,  |[pfc||,  is  bounded  away  from 
zero,  then  (2.10)  implies  that  jjp*|j  is  also  bounded  away  from  zero,  and  so  in  the  proof  of 
this  result  we  consider  only  those  iterations  in  which  ||pfc||  is  small. 

From  Lemma  3.1  and  A3  we  know  that  if  this  norm  is  small,  we  must  be  close  to,  x,  a 
stationary  point  of  NP,  and  in  that  case  we  can  use  the  results  from  Lemma  3.2  to  bound 
the  size  of  the  search  direction. 


Lemma  3.3.  There  exists  a  value  e'  >  0  such  that  i/||pjt||  <  e'  then  pk  is  a  minimizer  of  the 
QP  subproblem  and  the  correct  active  set  at  a  solution  has  been  identified.  Also,  ||p*||  =  0 
if  and  only  if  ik  is  a  first-order  KKT  point  of  NP. 


3  Global  convergence  results 
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Proof.  From  Lemmas  3.1  and  3.2,  there  exist  positive  constants  /Jspd  and  6°  such  that  if 
||p*||  <  and  pk  was  not  obtained  as  the  minimizer  of  the  QP  subproblem,  then 

HPk)  -  MPk)  >  fapd,  (3.1) 


and  from  the  continuity  of  ip,  there  exists  a  6  >  0  such  that  ||pjt  —  Pfe||  >  6. 
Define 

fit  = 


If  INI  <  fit ,  then 

INI  >  llPfc  -P*||  ~  INI  >\>fit- 

If  j|pfc||  >  (3°,  then  from  (2.10), 


\\Pk\\  > 


> 


fit_ 

fislp' 


and  thus  in  all  cases  the  final  point  obtained  has  norm  bounded  away  from  zero. 

If  pk  is  obtained  from  the  minimizer  of  the  QP  subproblem,  then  Lemma  3.1  can  be 
used  directly.  Assume  that  a  sequence  of  points  {x^}  exists  such  that  ||pfc||  — ►  0,  and  all  pjt 
are  obtained  as  the  solutions  of  the  corresponding  QP  subproblems,  but  the  active  sets  do 
not  correspond  to  the  one  at  a  solution.  By  extracting  a  subsequence  having  a  fixed  active 
set  and  taking  limits,  a  solution  for  the  original  problem  with  that  active  set  is  obtained 
(from  assumption  A3,  it  must  hold  that  the  multiplier  vectors  converge  to  the  multipliers 
at  the  limit  point),  contradicting  the  hypothesis.  Hence,  a  lower  bound  for  ||pjfc||  must  also 
exist  in  this  case. 

For  the  second  part  of  the  lemma  it  follows  from  the  previous  remarks  that  p*.  =  0  if 
ana  only  if  pk  is  a  solution  of  the  QP  subproblem.  Furthermore, 


Pk  =  0  is  a  solution  of  QP  gk  =  A jfp*,  p*  >  0,  c*  >  0,  pjck  =  0 

&  Xk  is  a  first-order  KKT  point  of  NP, 


completing  the  proof.  | 

Equivalence  of  sequences 

For  a  given  sequence  {i*},  the  next  result  establishes  the  equivalence  between  the  sequences 
{xk  -  x*}  and  {p*}. 

Lemma  3.4.  If  x*  denotes  the  solution  point  closest  to  Xk,  then  there  exists  a  constant 
Mp,  independent  of  k,  such  that 

\\*k-x*\\  <  A/p|N|.  (3-2) 

Proof.  The  proof  is  in  essence  the  one  given  for  Lemma  4.1  in  [GMSYV86b],  | 
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Descent  properties 

We  need  to  impose  some  condition  on  the  direction  pk  to  ensure  that  adequate  descent  can 
be  obtained  in  each  iteration.  To  be  precise,  it  is  necessary  to  satisfy  in  each  iteration  the 
bound  on  the  directional  derivative  given  in  step  (iii)  of  the  algorithm. 

The  next  lemma  shows  that  if  the  starting  point  for  the  QP  subproblem  is  selected  as 
indicated  in  Section  2,  the  search  direction  satisfies  a  certain  bound  that  will  be  shown  to 
imply  the  desired  result.  Recall  that  r k  was  the  quantity  introduced  in  Section  2  to  provide 
a  bound  for  the  norm  of  the  initial  point  p°,  and  that  its  most  relevant  property  for  the 
proofs  that  follow  is  its  relationship  to  c*.  -  s/j,  given  by  (2.6). 

Lemma  3.5.  There  exist  constants  0\  >0,0 2  >  0,  and  initial  points  for  the  QP  subproblem 
that  give  values  for  p*,  the  search  direction,  satisfying 

'l’(Pk)  =  Pk9k  +  \plHkpk  <  -0i\\pk\\2  +  &|N|.  (3.3) 


Proof.  In  the  proof  we  drop  the  subscript  corresponding  to  the  iteration  number.  Consider 
the  following  cases: 

(i)  p  is  the  minimizer  of  the  QP  subproblem.  Then, 

PTg  +  pTHp  -  pTATfi  =  -cTp  <  -p Tc~  <  ||/2||||c"|| 

<  ~\pTHp+  0nmu\\c-\\, 

where  /?„mu  >  0  is  a  bound  on  the  norm  of  the  QP  multipliers.  Note  that  from  the 
properties  of  H ,  pTHp  >  0,vh\\p\\2- 

(ii)  p  is  obtained  from  a  stationary  point  p  of  QP  that  is  not  a  minimizer.  Let  6°  be  the 
value  introduced  in  the  proof  of  Lemma  3.3.  One  of  the  following  three  cases  must 
apply: 

•  Assume  that  ||p||  >  6°  and  ||p-  p°||  <  %6°.  From  these  conditions,  ||p°||  >  ^6°. 

From  ||p°||  <  /ipcJ||r||,  assumptions  A1-A2  and  Lemma  2.1,  it  follows  that  ||p°||  <  K 
and 

rP(PO)<0nmgK  +  ^,vHK2  =  K. 

We  then  have  ip(p)  <  K ,  implying 

\{p  +  H-'g)TH(p  +  H'lg)  -  \gTH-xg  <  K, 


II P+  H  lg\\2  < 


2  A  0sl  7  -f  02mg 

Pin 


and  hence 
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giving  the  bound 


INI  <  flnmp  =  J^  + 


2K(3avH+(3l 


Using  this  property,  we  can  write 


Ml  <  ^||P°||. 


Defining  /?£  =  f3nrng  +  PivH0nmp,  we  have 


pts + pthp  <  am  <  <  2^^||r||, 


giving  finally 


1  ,  2/?|/3nmp/3pej  | 


0  <  -\p  Hp  + 


•  Assume  that  ||p-p°||  >  Let  p,  denote  the  tth  iterate  for  the  QP  subproblem  and 
0,  =  0(p,).  We  have 

0.-i  -  rPi  =  -<*&%  +  ptiHd,)  -  \ajdjlld,  =  djlld,  o,(l  - 
Summing  over  all  the  iterations  to  the  stationary  point,  and  letting  0  =  0(p)  we  get 

Vto  -  0  =  T,idJHdi  o,(l  -  jOj )  >  /WEill*||a«<(l  “ 

but  from  ||p  -  p°||  =  |(  53,  a«^«||  >  2^°’  f°r  at  Ieast  one  1  we  must  have 

oilWH  > 

where  m  is  a  bound  on  the  number  of  steps;  using  a,  <  1,  it  must  hold  that 

*"  -  *  -  0-“  (£)  -5)  ^ (£)  •  <3'4» 

From 

V>o  =  prp°  +  ip07"//?0  <  /32°||p°!|  <  /?pc^2°||r||  (3.5) 

we  can  derive  the  following  bound: 

’/><^<0o-7<  INI2  +  0PcS02|M| 

for  0  <  /?!  <  7/02mp- 
If  IIpII  <  <5°,  then  from  (3.1), 

00  -  0  >  0,pd, 

and  using  (3.5) 

0  ^  ~flspd  +  0pca02  Nil  <  — 0l||p||2  +  0pc»02||r||> 

where  0  <  /3,  <  / %pd/fimp •  I 
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Definition  of  the  penalty  parameter 

We  now  show  that  the  penalty  parameter  can  be  selected  in  such  a  way  that  the  initial 
descent  available  for  the  linesearch  is  sufficiently  large.  We  start  by  defining  a  scalar  p.  It 
will  be  shown  in  a  subsequent  lemma  that  4(0,  P)  satisfies  the  bound  given  in  step  (iii)  of 
the  algorithm. 

We  first  introduce  a  vector  b  and  a  nonnegative  scalar  0'2.  Their  values  should  be  selected 
as  follows: 


•  Define  pk  as  the  QP  multipliers  if  pk  is  the  minimizer  of  the  QP  subproblem;  otherwise 
define  fik  as  a  multiplier  estimate  satisfying  conditions  MC1-MC3. 

•  Define 

b={  P  if  PT9  +  PT(e  ~s)<  - pTHp , 

1  p  otherwise. 

•  Define 

0’2  =  max(0,/32), 

where 

02\\c  -  .s||i  =  pTg  +  \pTHp  +  bT(c  -  s). 


Note  that  0'2  is  bounded  since  from  Lemma  3.5  we  have, 


PT9  +  \pTHp  +  bT(c  ~  s)  <  pTg  +  pTHp  +  bT(c  -  s)  <  ( 02  +  ||<>||)||c  -  s||. 


The  value  for  0'2  presented  above  is  related  to  the  constant  introduced  in  (3.3),  while 
the  value  of  b  is  related  to  the  QP  multipliers  at  the  current  point.  For  the  purpose  of 
satisfying  the  bound  given  in  step  (iii)  of  the  algorithm,  b  can  be  taken  to  be  zero,  but  as 
it  will  be  seen  later,  it  plays  an  important  role  in  ensuring  that  the  penalty  parameter  is 
chosen  in  a  way  that  does  not  inhibit  superlinear  convergence. 

Define 


\\2\  -  p  -  b  +  0'2v\\ 

lie  -  <|| 


(3.6) 


where  «,  =  sign(cj  -  s,),  giving  vT(c  -  s)  =  ||c  -  s||i. 


Lemma  3.0.  For  all  p  >  pk,  where  pk  is  the  value  defined  in  (3.6), 

4(0, />)  <  -\plHkPk-  (3.7) 


Proof.  Again,  we  drop  the  subscript  corresponding  to  the  iteration  number. 

We  start  by  introducing  an  expression  for  <^'(0).  To  derive  it,  consider  first  the  gradient 
of  La  with  respect  to  x,  X  and  s, 


VL„(z,A,s) 


g(x)  -  A(i)rA  pA(x)T(c(x)  -  s) 
~(c(x)  -  s) 

X  -  p(c(x)  -  s) 


(3.8) 
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It  follows  that  4>'(Q)  is  given  by 

<£'(0)  =  pTg  -  pTATX  +  ppTAT(c  -  s)-(c-  s)T£  +  XTq  -  pqT(c  -  s) 

=  PT9  +  (2A  -  p)T(c  -s)-  p\\c  -  s||2,  (3.9) 

where  g,  A ,  and  c  are  evaluated  at  x. 

From  (3.9)  and  (3.6), 

<j>'(0,p)  =  pTg  +  (2A  -  p)T(c  -  s)-  || c  -  s|||]2A  -  p  -  b  +  (3'2v\\ 

<  pTg  +  bT(c  -  s)  -  (i'2 vT(c  -  s) 

<  - \pTHp , 


completing  the  result.  | 


An  immediate  consequence  of  (3.7)  and  the  properties  of  Hk  is  the  following  bound  on 
the  directional  derivative: 

44(0)  <  -^H\\pk\\\  (3.10) 


for  (3h  <  /3avH. 

The  strategy  for  the  selection  of  the  penalty  parameter  pk  is  to  define  its  value  to  satisfy 
(3.7),  while  remaining  small  enough  to  be  bounded  by  a  multiple  of  p.  An  example  of  a 
selection  rule  having  these  properties  is  as  follows. 


Let 


Pk- 1 

max(pfc,2pfc_i) 


\{ <t>'{0, pk-i)  <  -\p{Hkpk, 

otherwise, 


(3.11) 


where  pk  is  defined  as  in  (3.6).  Then,  for  any  iteration  ki  in  which  the  parameter  needs 
to  be  increased,  it  holds  that  pk(  >  2pk,_l-  It  follows  from  this  result  and  (3.6)  that  the 
penalty  parameter  goes  to  infinity  if  and  only  if  its  value  is  increased  in  an  infinite  number 
of  iterations. 

For  the  rest  of  this  paper,  the  symbol  pk  will  denote  a  value  of  the  penalty  parameter 
satisfying  pk  <  Pk  <  Kpk i  for  some  constant  K  >  1. 


Proof  of  global  convergence 

The  proof  of  global  convergence  is  based  on  the  fact  that  the  decrease  in  the  value  of  the 
merit  function  in  each  iteration  is  bounded  away  from  zero  by  a  sufficiently  large  value, 
related  to  the  norm  of  the  search  direction.  This  fact  is  implied  by  the  existence  of  certain 
bounds  on  the  rate  of  growth  for  the  penalty  parameter,  introduced  in  the  following  lemma. 
The  notation  ki  is  used  in  all  that  follows  to  indicate  iterations  at  which  the  value  of  the 
penalty  parameter  needs  to  be  modified. 


Lemma  3.7.  For  any  iteration  ki  in  which  the  value  of  p  is  modified, 


PfcJkfc,  -  sfc,||  <  N 
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and 

PkilKII2  <  N, 

for  some  constant  N . 


Proof.  All  quantities  in  the  proof  refer  to  iteration  ki ,  and  so  this  subscript  is  dropped. 

From  the  boundedness  of  P'2,  Lemma  2.1,  the  definition  of  6,  and  condition  MCI  on  the 
multipliers,  there  must  exist  a  constant  N i  such  that 


||2A  -  p-b  +  P2v\\  <  Ni, 

and  from  the  definition  of  p  and  the  condition  that  p  has  to  be  selected  as  a  finite  multiple 
of  p , 

p||c-  s||  <  N\. 

For  the  second  part,  using  Lemma  3.5  (we  add  the  term  bT(c  -  s)  using  the  boundedness 
of  ||6||),  we  can  write  after  some  algebraic  manipulation 

*'(0)  =  PTg  +  (2A  -  pf(c  -s)-  p\\c  -  s|j2 

<  -\pTHp  -  Pi\\p\\2  +  {2\-p-b  +  P2v)T(c  ~s)~  p(c  -  s)T(c  -  s). 

Since  we  have  ,p~)  >  - \prHp ,  where  p~  is  the  value  of  p  at  the  previous  iteration,  it 
follows  that 

Pi\\p\\2  <  (2A  -  p  -  b  +  P2vf{c  -  s) 

<  ||2A  -  p.  -b  +  P2v\\  ||c  -  s||. 

We  reorder  terms  to  obtain 


Ik  -  -II  >  Ih 


IIpII2 

||2A  -  p-b  +  P2v\\‘ 


(3.12) 


Multiplying  both  sides  by  p  and  using  the  same  arguments  as  in  the  first  part  of  the  lemma 
yields 

p||p||2  <  n2, 

completing  the  proof.  | 

The  convergence  proof  follows  along  lines  similar  to  those  presented  in  [GMSW86b].  The 
following  theorem  relies  on  results  presented  in  this  reference,  that  hold  with  only  minor 
modifications  for  the  algorithm  considered  in  this  paper. 


Theorem  3.1.  The  algorithm  has  the  property  that 

Jim  |jxfc  -  i*||  =  0. 

fc— *00 
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Proof.  It  follows  from  Lemma  3.4  that  it  is  sufficient  for  the  proof  of  the  theorem  to  show 

lim  ||Pfc||  =  0.  (3.13) 

fc— *00 

If  ||pfc||  =  0  for  any  k,  the  algorithm  terminates  and  the  theorem  is  true.  Hence  we  assume 
that  ||pfc||  ^  0  for  any  k. 

Consider  first  the  case  when  there  is  no  upper  bound  on  the  penalty  parameter.  The 
following  result,  given  as  Lemma  4.6  in  [GMSW86b],  and  as  Lemma  3.6.3  in  [Pr89],  holds 
for  the  algorithm: 

Ph  Yj  \\akPk\\2<M,  (3.14) 

k=k( 

where  M  is  a  positive  constant.  It  also  holds  that  there  exists  a  uniform  lower  bound  on 
ak,  0  <  a  <  atk,  (Lemma  4.9  in  [GMSW86b],  and  Lemma  3.6.6  in  [Pr89]).  From  these 
properties,  (3.14)  implies  that  for  any  6  >  0  we  can  find  an  iteration  index  K  such  that 

||p* H  <  f>  for  k  >  K, 

which  implies  that  ||pfc||  — >  0  as  required. 

In  the  bounded  case,  we  know  that  there  exists  a  value  p  and  an  iteration  index  K  such 
that  p  —  p  for  all  k  >  K.  We  consider  henceforth  only  such  values  of  k. 

The  proof  is  by  contradiction.  We  assume  that  there  exists  c  >  0  and  an  infinite 
subsequence  {ki}  such  that  ||pfc,||  >  c  for  all  i.  Consider  only  indices  i  such  that  k,  >  K. 
Every  iteration  after  K  must  yield  a  strict  decrease  in  the  merit  function  because  the 
boundedness  of  the  steplength  implies 

4>{a)  -  0(0)  <  <ra0'(O)  <  -\oa^H\\p\\2  <  0. 

The  adjustment  of  the  slack  variables  s  in  step  (ii)  of  the  algorithm  can  only  lead  to  a  further 
reduction  in  the  merit  function,  as  L  is  quadratic  in  s  and  the  minimizer  with  respect  to 
s,  is  given  by  c,  -  A ,/p.  From  the  fact  that  the  penalty  parameter  is  not  modified,  for 
iterations  from  the  subsequence  we  have 

<p(xk,+ j)  -  4>{xk)  <  4>(xk,+ 1)  -  0(xfc)  <  -Acra/3wc2. 

Therefore,  since  the  merit  function  with  p  =  p  decreases  by  at  least  a  fixed  quantity  at 
every  step  in  the  subsequence,  it  must  be  unbounded  below.  But  this  is  impossible,  from 
assumptions  Al,  A2  and  Lemma  2.1,  so  (3.13)  must  hold.  | 

Once  the  global  convergence  of  the  algorithm  has  been  established,  the  next  step  is 
to  show  that  the  multiplier  estimate  A*  also  converges  to  the  desired  value.  The  result 
presented  below,  given  as  Theorem  4.2  in  [GMSW86b],  implies  that  the  convergence  of  the 
multiplier  estimates  is  a  consequence  of  the  global  convergence  of  the  algorithm,  and  the 
facts  that  the  multiplier  estimates  are  bounded  in  norm,  and  the  steplength  is  bounded 
away  from  zero. 

Corollary  3.1.  Let  X*  denote  the  multiplier  vector  at  x*.  Then 

lim  || Afc  -  A*  ||  =  0. 
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4.  Rate  of  convergence 

Under  additional  assumptions  we  shall  show  that  the  algorithm  converges  at  a  superlinear 
rate.  In  order  to  do  this  we  first  prove  a  number  of  results  on  the  rate  of  growth  of  the 
penalty  parameter  p. 

Lemma  4.1.  If  there  exists  an  infinite  subsequence  of  iterations  {fc/}  at  which  the  penalty 
parameter  is  increased,  then 

lim  pk,\\ck,  -  ^ki  II  =  0 

/— *oo 

and 

lim  p/c,||Pfc,||2  =  0. 

/— ►  OO 

Proof.  We  drop  the  subscript  ki  in  what  follows.  From  definition  (3.6)  and  boundedness 
of  the  ratio  p/p , 

p\\c  -  <s||  <  2||2A  -p-b  +  fi'2v\\, 
and  from  the  definition  of  b  after  Lemma  3.6, 

bk,  -  A*. 

As  the  QP  multipliers  satisfy  pTg  +  pTHp  =  -cTfi,  and  for  p  large  enough  p  is  obtained  as 
the  solution  of  the  QP  subproblem,  b  eventually  satisfies 

PT9  +  bT(c  -  s)  <  —pTHp, 

implying  that  we  can  take  (3'2  =  0  in  (3.6). 

We  can  now  use  Corollary  3.1  to  conclude  that 

lim  ||2Afc,  -  pk,  -  bk,  +  fiLn, ||  =  0 

/— ►OO  1 

and 

lim  pk, ||cfc,  —  5*, ||  =  0.  (4.1) 

/— ►OO 

Finally,  from  (3.12)  and  (4.1)  we  have 

lim  pfc(i|pfc(||2  =  0, 

1—+00 

completing  the  proof.  | 

Lemma  4.2.  If  there  exists  an  infinite  subsequence  {fc/},  then 

Um  Pk,  (<Pk,(Pk,)  -  4>k,+  ,(Pk,j)  =  0. 


4  Rate  of  convergence 
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Proof.  To  simplify  the  notation  we  shall  use  the  subscripts  0  and  K  to  denote  quantities 
associated  with  the  iterations  k\  and  ki+\  respectively.  Thus,  the  penalty  parameter  is 
increased  at  xo  and  xr  and  remains  fixed  at  po  for  iterations  1, . . . ,  K  -  1. 

From  the  boundedness  of  ||A||  (Lemma  2.1),  and  the  fact  that  po  <  p*.-,  we  have 

A>|'\)(co  -  so)|  <  2||A0|I  polko  -  sol!  “* ’  0> 

Po\^Ack  ~  s*)|  <  2||A* l|  Pk\\ck  ~  Sjc  1!  0) 


and  from  Lemma  4.1  we  have 


Po(<t>o  -  4>k )  -  Po(Fo  -  Fk)  — *■  0. 


(4.2) 


Using 


PqCqPo  <  PoA'IlPoll2  +  po(c0  -  s0)r(2A0  -  p0). 


(4.3) 


we  have 

PoA'HpoI!2  +  Po(c o  -  s0)r(2A0  -  po)  >  po($P o  >  Po(co  -  »o)TP o-  (4.4) 

Using  again  Lemma  4.1,  from  (4.4)  and  assumption  A3,  implying  the  boundedness  of  ||pol|> 
we  get 

PoCofio  -*•  0.  (4.5) 

From  (2.6)  (keeping  the  same  notation), 

-  PoclPo  <  PoCkTPo  <  PoIIpoIIH'ca-  -  sK-||  —  0.  (4.6) 

We  can  again  use  Lemma  4.1  to  obtain 


p0o(max(||po||2,||pK||2))  -+  0. 


(4.7) 


From  (4.2),  (4.5),  (4.6)  and  (4.7)  we  obtain 


Po(0o  -  4>k)  -*  0, 


giving  the  desired  result.  | 

Lemma  4.3.  For  general  iterations  k, 

Jim  pk\\pk\\2  =  0. 

AC— *00 

Proof.  We  use  the  same  notation  as  lemma  4.2. 

If  p  is  bounded,  the  result  follows  from  Theorem  3.1.  If  p  is  increased  in  an  infinite 
subsequence  of  iterations,  then  for  0  <  k  <  K  -  1,  property  (2.5a)  imposed  by  the  choice 
of  <*fc,  and  the  fact  that  the  penalty  parameter  is  not  increased,  imply  that 


<t>k  ~  1  >  -OGkfik- 


(4.8) 
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I 

I 

We  can  then  write 

K- 1 

2°Ph  Y1  a-t!|P*H2  <  ^  -  $ K ■ 

Jt=0 

Rearranging  this  expression  and  using  the  property  that  0  <  a  <  a*,  we  obtain 

K-l 

\oPHot  llPfell2  <<fo-<t>K,  (4.9) 

k=o 

and  the  result  follows  from  Lemma  4.2.  | 


Lemma  4.4.  For  genual  iterations  k, 

lim  pk\\ck  -  Sfc||  =  0. 
»00 


Proof.  If  p  is  bounded  the  result  follows  from  c*  >  0,  A*  >  0,  X*Tc*  —  0,  Corollary  ??, 
Lemma  3.1  and 

c,  -  Si  =  min(c,,  — ). 

P 

If  p  is  increased  in  an  infinite  subsequence  of  iterations,  consider  two  cases: 

(i)  If  t  is  such  that  c*  >  0,  then  A*  =  0  and  as 


p\ci  -  s,|  =  |min(pc,,A,)|, 


from  the  convergence  of  the  multiplier  estimates,  eventually  p\c{  —  s,|  =  |A,|  — ►  0. 

(ii)  For  those  i  such  that  c*  =  0,  implying  A*  >  0,  consider  iteration  indices  large  enough 
so  that  the  correct  active  set  is  identified,  implying  ajp  +  c*  =  0.  Then,  from  the 
Taylor  series  expansion  for  c  and  the  boundedness  of  the  steplength, 

Ci  =  cj  +  a0ajp  +  0(||a0po!|2)  =  (1  -  ao)c,  +  0(||po||2). 

Recurring  this  relationship  for  the  fcth  step  between  k  =  0  and  k  -  K  we  get 

k-l  k-l 

Pk(ck)i  =  po(cfc),  =  po  ”  ai )(**>)«'  +  PoO{l>2  INI2)’ 
j= o  j= o 


but  as  0  <  Qj  <  1  we  obtain 


k-l 


Pfc|(cfc).|  <  po|(co)i|  +  poO(Yl  llPjll2)- 

i- o 


(4.10) 


From  Lemma  4.1  we  must  have  that  po|(co)»|  — »  0,  and  using  (4.10)  and  Lemma  4.3, 

P*l(c*).l  —  0. 
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This  completes  the  proof.  | 

Lemma  4.5.  For  large  enough  k, 

nlsk  =  0. 


Proof.  Assume  k  large  enough  so  that  the  correct  active  set  has  been  identified. 

(i)  If  i  is  such  that  c*  >  0,  from  condition  MC3  on  the  multipliers,  pki  =  0. 

(ii)  If  i  is  such  that  c*  =  0,  then,  from  strict  complementarity,  A*  >  0.  Also,  from 
Lemma  4.4,  pk((ck)i  -  ( s*)t- )  =  min(p*(cfc)t,(Ajfc),)  ->  0,  so  for  large  enough  k, 
Lemma  3.1  will  imply  pk(ck)i  <  (A and 

(sk)i  =  max(o,(c*).  -  =  0, 

v  Pk  ' 


proving  the  result.  | 

To  prove  that  the  algorithm  converges  superlinearly  it  is  necessary  to  assume  that  Hk 
converges  to  an  approximation  of  Vxx£( a:*,  A*)  in  some  sense.  We  shall  also  introduce  a 
stronger  condition  on  the  Lagrange  multiplier  estimate. 

Define  Wk  as  Wk  =  VxxL(xfc,  A*).  We  impose  the  following  condition  on  Hk: 

A7.  Following  Boggs,  Tolle  and  Wang  [BTW82],  we  assume 

II Zl(Hk  -  Wk)Pk ||  =  o(||pfc||), 

where  Zk  is  a  basis  for  the  null  space  of  Ak  that  is  bounded  in  norm  and  has  its 
smallest  singular  value  bounded  away  from  0. 

The  proof  proceeds  by  first  showing  that  the  sequence  {x*  +  pk  -  x*}  converges  su¬ 
perlinearly,  and  then  proving  that  a  steplength  of  one  is  eventually  attained.  The  results 
presented  on  bounds  for  the  growth  rate  of  the  penalty  parameter  allow  us  to  obtain  an 
asymptotic  expansion  for  the  quantities  involved  in  the  linesearch  termination  criterion. 

We  want  to  prove  that  condition  (2.4)  is  satisfied  for  all  k  large  enough.  We  show  in  the 
following  lemma  that  the  satisfaction  of  (2.4)  is  directly  related  to  the  asymptotic  properties 
of  Tjt  =  pj{gk  -  Ajpk)  +  pJWkpk.  In  what  follows,  the  absence  of  an  argument  indicates 
values  at  xk,  and  an  argument  of  1  will  indicate  values  at  xk  +  pk. 

Lemma  4.6.  The  following  relationship  holds: 

<Afe(i)-^  =  ^  +  in+o(iiPfc|i2). 
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Proof.  From  (2.1)  we  have 

<KD-<t>  =  F(l)-F  -  pT(c(l  )-s-q)  +  \T(c-s) 

+  |(c(i)  -  5  -  q)T(c(  1)  -  S  -  q)  -  |(c  -  s)r(c  -  s). 

Using  the  corresponding  Taylor  expansions  around  x^, 

Ci{  1)  -  Si  -  qi  =  lpTV2c,p+  o(||p||2), 


we  obtain 


<K1)  -  <t>  =  gTP  +  \pTV2Fp  -  ^E.A.p^c.p  -  {^/vVp 

+  Ar(c  -  s)  +  |D(pTV2c,P)2  -  f(c  -  s)r(c  -  s)  +  0(||p||2).  (4.11) 

From  Lemma  4.3  and  (4.11), 

4>{\)-4>  =  <?/  +  i(pTlFp  +  2£r(c-s)  +  p(c-s)7(c-s))  +  o(||p||2) 

=  \<t>'  +  \  (pTWp  +  PT9  +  PT(c  -  «))  +  o(||pi|2) 

=  2^'  +  2  (pTWp  +  pT{g  -  4Tp))  +  o(||p[|2), 
completing  the  result.  | 

The  assumptions  made  imply  the  superlinear  convergence  of  the  sequence  {xjt  +  p*  —  a:*}. 
The  following  lemma  corresponds  to  Theorem  3.1  in  [BTW82]. 

Lemma  4.7.  Under  assumptions  A1-A7,  and  conditions  MC1-MC3, 

Ik*  +  Pk  ~  z*H  =  o(lk*  -  **||).  (4-12) 

The  main  result  of  this  section  is  given  in  the  next  theorem,  where  it  is  shown  that,  if 
condition  MC2  is  replaced  by  a  stronger  condition,  then  after  a  finite  number  of  iterations  a 
steplength  of  one  is  taken  for  all  iterations  thereafter,  implying  that  the  algorithm  achieves 
superlinear  convergence. 

The  new  condition  is 

MC2\  ||/xjt  —  A* ||  =  o(||x*  -  i*||). 

It  is  possible  to  prove  Theorem  4.1  without  the  need  to  strengthen  the  conditions  on 
the  multipliers.  It  is  shown  in  [Pr89]  that  there  exists  a  constant  M  such  that  if  p*  >  M, 
condition  MC2  is  sufficient. 

Theorem  4.1.  Under  assumptions  A1-A7,  and  conditions  MCI,  MC2’  and  MC3,  the 
algorithm  converges  superlinearly. 


4  Rate  of  convergence 
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Proof.  As  in  Powell  and  Yuan  [PY86],  observe  that  the  continuity  of  second  derivatives 
gives  the  following  relationships: 

F(x  +  p)  =  F(x)  +  i  (<?(*)  +  g{x  +  p))Tp  +  o(\\p\\2) 

c(x  +  p )  =  c(x)  +  l(A(x)  +  A(x  +  p))p  +  o(||p||2). 

From  the  Taylor  series  expansions  we  have 

F(x  +  p)  =  F(x)  +  g(x)Tp  +  \pTV2F{x)p  +  o(||p||2) 

Ci(x  +  p)  =  c,(i)  +  a,(x)Tp  +  |pTV2Ci(x)p  +  o(j|p||2), 

and  since  (4.12)  implies  g(x  +  p)  =  g*  +  o(||p||),  cq(x  +  p)  =  a*  +  o(||p||),  we  get 

pTV2Fp  =  (g*  -  gfp  +  o(||p||2) 
pTV2c,p  =  (a*  -  a,)Tp  +  o(||p||2). 

Given  that  A,  pTV2c,p  =  p,  pTV2c,p  +  o(j|p||2),  we  must  have 

pTWp  =  pT(g*  -  A*Tp)  -  pT(g  -  ATp)  +  o(||p||2).  (4.13) 

Condition  MC2’  implies  pT(g*  -  A*Tp)  =  o(j|p|)2),  and  from  (4.13), 

pTWp  +  pT{g  -  ATp)  =  o(||p||2).  (4.14) 

From  Lemma  4.6  and  (4.14), 

0(l)-<A(O)  =  ^/(O)  +  o(||p||2), 


but  from  (3.10)  condition  (2.4)  is  eventually  satisfied,  and  we  have  Xk+i  =  x*  +  pjt  for  all  k 
large  enough.  In  this  case,  from  (4.12), 


i.e. 


||**+i  ~  **|| 

II**-**  II 

superlinear  convergence,  completing  the  proof. 


=  0, 

I 


Boundedness  of  the  penalty  parameter 

The  last  result  in  this  section  shows  that,  if  condition  MC2’  is  replaced  by  a  slightly 
stronger  condition,  the  penalty  parameter  needs  to  ’  e  modified  in  at  most  a  finite  number  of 
iterations  (and  consequently  it  remains  bounded).  The  criterion  presented  will  be  satisfied, 
for  example,  by  the  least-squares  multipliers  computed  at  n  +  p*. 

Theorem  4.2.  //  the  multiplier  estimates  pk  in  the  algorithm  satisfy 

||pit  -  A*||  =  0(||xt  +  pk  -  x*||),  (4.15) 

the  penalty  parameter  is  only  increased  a  finite  number  of  times,  and  there  exists  a  constant 
M  such  that  pk  <  M  for  all  k. 
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Proof.  Let  K  be  an  iteration  index  such  that,  for  all  iterations  k  >  A',  pk  is  determined 
as  the  minimizer  of  QP  and  =  1.  We  now  consider  iterations  having  k  >  K . 


9kPk  +  plHkPk  =  PkAknk  =  -CfcTTfc  <  ~(ck  -  sk)Tirk,  (4-16) 

where  nk  are  the  QP  multipliers  at  iteration  k.  From  (3.9),  (4.16)  and  the  fact  that  a  unit 
steplength  is  accepted,  it  follows  that 

0*(°)  =  9lPk  +  (2A*  -  Hk)T(ck  -  sk)  -  /»*||c*  -  s*||2 

<  -  p[Hkpk  +  ||2pfc_i  -  Hk  -  fffc||||cfc  -  sk ||  -  pk\\ck  -  sfc||2. 

If  (4.15)  holds,  from  the  properties  of  Hk  and  j|iTfc  —  A*j|  =  0(||pjc||)  we  must  have 

||2p*_i  -  pk  -  x*||  <  MplHkpk 

for  some  positive  constant  M . 

We  can  then  write 

||2/x/t— i  -  Pfc  -  TTfcllllcfc  -  Jkll  <  \p[Hkpk  +  %M\\ck  -  Sjtjl2, 


implying  that 

M 0)  <  - \pTkHkpk  +  i\M  -  pk)\\ck  -  Sfcll2. 

From  this  inequality  it  follows  that  if  pk  >  \M ,  condition  (3.7)  will  be  satisfied,  and  the 
penalty  parameter  will  not  need  to  be  increased.  Given  that  we  are  using  the  rule  for 
updating  pk  described  after  Lemma  3.6,  it  must  hold  that  pk  <  \KM .  I 

Note  that  the  lemma  does  not  require  any  conditions  on  how  well  Hk  approximates  Wk. 

5.  Other  Merit  Functions 

Several  merit  functions  have  been  proposed  and  analyzed  in  the  literature  (a  review  can 
be  found  in  Powell  [Po87]).  The  question  arises  if  the  convergence  results  using  early 
termination  in  the  solution  of  the  QP  subproblem  depend  on  our  specific  merit  function,  or 
if  they  are  fairly  independent  of  this  choice.  We  shall  show  in  this  section  the  choice  of  merit 
function  is  not  critical.  What  we  present  is  how  to  adapt  our  SQP  algorithm  to  the  use  of 
other  merit  functions  rather  than  examine  other  methods  explicitly  to  see  if  the  particular 
QP  subproblem  posed  and  the  manner  the  search  is  performed  can  be  adapted  to  the  use  of 
an  incomplete  solution.  For  example,  we  still  perform  a  search  in  the  x  and  A  spaces.  Slack 
variables  do  not  appear  in  the  merit  functions  we  shall  consider,  consequently  the  search 
in  the  space  of  the  slack  variables  is  no  longer  required.  Because  the  freedom  to  choose  the 
multiplier  estimates  is  relaxed,  our  algorithm  can  be  made  to  mimic  other  algorithms.  For 
example,  we  are  free  to  choose  all  the  multiplier  estimates  to  be  zero  making  the  search  in 
the  multiplier  space  specious. 

We  have  selected  as  examples  the  study  of  two  particular  merit  function.  The  first  one 
corresponds  to  a  class  of  merit  functions  that  includes  among  others  the  i\  merit  function 
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analyzed  in  Han  [Han76],  Byrd  and  Nocedal  [BN88]  and  Burke  and  Han  [BH89].  This 
general  merit  function  takes  the  form: 

0(x,  A)  =  F(x)  +  XTc~(x)  +  p||c"(x)||,  (5.1) 

where  an  lv  norm  (1  <  p  <  oo)  is  used,  and  c"(x)  =  max(0, -c,(x)).  The  second  merit 
function  we  consider  is 


<t>(x,\)  =  F(x)  +  XTc  (x)  +  ±p||c  (ar)||l-  (5.2) 

This  merit  function  has  been  studied  among  others  by  Powell  and  Yuan  [PY86].  Unlike  the 
algorithm  defined  in  [PY86],  we  do  not  explicitly  define  the  form  of  the  multiplier  estimates 
although  the  one  used  in  [PY86]  satisfies  the  criteria  on  our  estimates. 

We  still  assume  A1-A6  hold  for  the  problem.  However,  the  multiplier  estimate  used, 
(ik,  is  only  required  to  satisfy  MCI  when  the  merit  function  (5.1)  is  used.  This  condition 
is  trivial  to  satisfy.  For  example,  we  may  choose  Ao  =  0  and  £k  =  0.  Such  a  choice  reduces 
(5.1)  to  the  well-known  /j  merit  function  and  our  algorithm  becomes  very  similar  to  that 
analysed  in  [Han76].  When  (5.2)  is  used,  we  assume  conditions  MCI  and  MC3  hold. 
Again  it  is  relatively  trivial  to  satisfy  these  conditions.  We  have  also  assumed  in  the  proofs 
that  Ao  >  0  and  pk  >  0. 

The  criteria  given  in  section  2  for  the  choice  of  steplength  at  assumes  the  merit  function 
has  continuous  first  derivatives.  This  property  does  not  necessarily  hold  for  the  merit 
functions  under  consideration.  Therefore  we  use  the  following  criteria  for  determining  a 
value  at.  The  steplength  at  is  required  to  satisfy 

4>(otk)  =  <t>(xk  +  OtkPk,  *k  +  OtkZk)  <  m  +  T)Qk&k,  (5.3) 

where  At  =  gkPk  +  (£/c  -  ^k)Tc~(xk)  -  Pfc||c-(*t)||»  and  'n  addition  must  either  satisfy 

Ok  >  li  >  0  (5.4) 

or 

Ok  >  7u ok  and  <p(ak)  >  <t>( 0)  -I-  cdtA k,  (5.5) 

where  0  <  7/  <  7«  <  1,  0<q<a<l  and  ak  >  0.  For  a  discussion  of  this  criteria  and 
the  existence  of  ak  see  Calamai  and  More  [CM87].  Our  preference  for  the  criteria  given  in 
section  2  is  based  on  our  belief  that  in  practice  they  lead  to  a  better  choice  of  ak ■  In  the 
definition  of  our  algorithm  we  could  have  used  other  steplength  criteria  without  impacting 
the  convergence  properties. 

The  following  basic  relationships  will  be  used  to  establish  the  convergence  results, 

c~ (x  +  ap)  <  | C| ( x  +  ap)  -  c,(x)  -  aajp\  +  (c,(x)  +  aafp)~  (5.6a) 

(c,(x)  +  aajp)~  <  (1  —  a)c“ (x)  (5.6b) 

-u>TAp  <  -||c~(x)j|,  (5.6c) 

where  u>TAp  represents  an  element  of  chp(0),  the  subdifferential  of  <p(<*)  =  ||c~(x  +  ap)||  at 
0.  The  elements  of  w  take  values  in  [0,  lj  and  are  0  whenever  c,(x)  >  0.  To  simplify  the 
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notation  in  what  follows,  we  introduce  a  diagonal  matrix  ft  formed  from  the  elements  of  u 
as  ft  =  diag(u>,). 

Consider  now  the  case  when  <f>  has  been  defined  from  (5.1).  From  our  assumption  that 
>  °, 

\j£lk{AkPk  +  <*)  >  0 

for  all  k.  It  follows  from  this  inequality  and  the  relationships  given  in  (5.6)  that 
^(0)  =  A  +  (kc~(xk)  -  \kSlkAkpk  -  pkulAkPk  <  A k- 
We  select  pk  such  that 

A fc  <  -\pTkHkVk.  (5.7) 

This  rule  is  analogous  to  the  ones  used  in  Byrd  and  Nocedal  [BN88],  and  Burke  and  Han 
[BH89]. 

The  first  step  is  to  establish  that  such  a  value  of  p  exists.  From  Lemma  3.5  we  have 

A,  <  -ftllmll2  +  AIM  -  hflH, ,rt  +  lift  -  AtlllM  -  (Hl<f  II. 

and  defining  pu  =  fi2  +  3/?„mu,  for  any  value  p  >  pu  condition  (5.7)  is  satisfied  for  any  k. 
This  result  also  shows  that  the  value  of  p  will  remain  bounded  in  the  algorithm. 

Theorem  5.1.  The  algorithm  modified  to  use  the  merit  functions  (5.1)  converges  globally. 


Proof.  Given  the  results  from  Section  3,  it  is  enough  to  show  that  ||pjt|i  — *•  0. 

As  p  cannot  grow  without  bound,  any  strategy  for  increasing  p  by  a  finite  quantity  when 
required  to  do  so  must  imply  that  there  exists  an  iteration  value  K  such  that  pk  =  pK  for 
all  Ar  >  1\.  We  consider  only  iterations  of  this  form.  For  k  >  K , 

<£(<**)  -  4>(<*k- 1)  <  Ofc^Afc  <  -(T0»vH<Xk\\Pk\\2- 

From  the  boundedness  of  <f>,  it  follows  that  «fc||pfcll2  — ^ ►  0. 

^  llPfcll  ~ ^  convergence  follows  from  Lemma  3.1.  Otherwise,  if  for  a  subsequence 
IIp*II  >  € i  we  must  have  ak  — *  0  along  the  subsequence,  and  from  the  termination  conditions 
for  the  linesearch,  a*  — >  0. 

From  the  definition  of  the  merit  function  (5.1),  (we  drop  the  index  k  in  the  following 
relationships) 

4>(a)  -  <£(0)  =  agTp  +  A7'(c_(d)  -  c"(0))  +  a^Tc~(a)  -  dp||c“(0)|| 

+  (F(d)  -  F(0)  -  agTp)  +  p(||c-(d)||  -  (1  -  «)||e~(0)||). 

For  the  last  term,  from  (5.6a)  and  (5.6b),  it  follows  that 


II c  (d)||  -  (1  -  d)||c  (0)||  <  ||c(d)  -  c(0)  -  dAp||, 
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and  from  this  result  we  must  have 

4>(a)  -  d>(0)  <  agTp+  Ar(c~(d)  -  c"(0))  +  d(rc_(d)  -  dp||c"(0)|| 

+  (F(d)  -  F( 0)  -  dgTp)  +  p||c(d)  -  c(0)  -  aAp\\. 

If  we  use  again  (5.6a)  and  (5.6b)  on  the  terms  associated  with  the  multiplier  estimates 
(given  that  by  assumption  A  +  d(  >  0),  we  obtain 

4>(d)  -  4>{0)  <  ctgTp  +  5Zt(A,-  +  d£t)r|c,(d)  -  Cj(0)  -  aafp\  +  (1  -  d)Arc"(0) 

-  Arc"(0)  +  d(I  -  d)^Tc-(0)  -  dp|(c-(0)(|  +  0((jdp||2). 

Simplifying  this  expression  gives 

4>{a)  -  <}>{Q)  <  d(gTp  +  ((  -  A)Tc~(0)  -  p||c~(0)||)  +  d2||c~(0)||  ||f  ||  +  0(||dp||2).  (5.8) 

Replacing  this  bound  in  the  linesearch  termination  condition  implies 

0  <  (1  -  <r)dA  +  d2||c-(0)||  IKK  +  0(||dp||2).  (5.9) 

If  in  this  equation  we  use  the  bound  A  <  - /?*»// ||p||2  and  we  take  into  account  that  ||p||  is 
bounded  away  from  zero,  by  taking  limits  along  the  subsequence  we  obtain 

0  <  -(1  -(T)P,vHt\ 

but  this  is  not  possible,  so  we  must  have  ||p*||  — ►  0  for  the  whole  sequence.  | 

We  now  consider  the  second  merit  function  (5.2).  The  subgradient  along  the  search 
direction  at  (xk,Xk)  is  given  by 

#(0)  =  glpk  +  (kc~(Xk)  -  ffikAkPk  ~  Pkc~ixk)TAkPk  <  A*, 

where  we  have  defined 

A*  =  g{pk  +  (fr  -  A* )Tc~(xk)  -  ^||c_(x*)||2, 
and  we  have  made  use  of  A*  >  0  to  have 

(fit A*  +  PkC^)T{Akpk  +Ck)  >  0. 

In  this  case  it  is  not  immediately  evident  that  pk  remains  bounded.  The  convergence 
proof  we  give  is  similar  to  the  one  introduced  in  Section  3.  The  definition  of  p  given  in  that 
section  will  be  preserved,  replacing  only  c  -  s  by  c~. 

Theorem  5.2.  The  algorithm  modified  to  use  the  merit  function  (5.2)  converges  globally. 
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Proof.  Again,  from  the  results  in  Section  3  it  is  enough  to  show  that  ||p*||  — ►  0,  although 
now  we  need  to  consider  two  cases. 

If  pk  remains  bounded,  then  as  in  the  proof  of  Theorem  5.1,  we  must  have  that  o*||p;b||2  — ► 

0. 

If  ||p*||  — ►  0,  convergence  follows  from  Lemma  3.1.  Otherwise,  if  for  a  subsequence 
||;>*||  >  c,  we  must  have  a*  — ►  0,  and  from  the  termination  conditions  for  the  linesearch, 

Q*  -»  0. 

The  argument  when  p  is  bounded  is  similar  to  that  given  in  Theorem  5.1.  From  (5.6a) 
and  (5.6b),  we  also  have  (we  drop  the  index  k  in  the  following  relationships) 

<j>(a)  -  <j>( 0)  <  agTp+ XT(c~(a)  -  c"(0))  +  a(Tc~(a)  -  p(a  -  id2)j|c~(0)||2 

+  p||c(a)  ~  c(0)  -  aAp||(ij|c(d)  -  c(0)  -  aAp\\  +  ||(c(0)  +  <5Ap)~||) 

+  (F(a)-F(0)-agTp), 

and  again  using  (5.6a)  and  (5.6b)  on  the  terms  associated  to  the  multiplier  estimates,  we 
obtain 

'/■(o)  -  ■?(<!)  <  a{gTV  +  ({  -  A)V(O)  -  p||c-(0)|p) 

+  a2|[c-(0)l|  (||f  [I  +  irf|c-(0)||)  +  0(||6p|f).  (5.10) 

Replacing  this  bound  in  the  linesearch  termination  condition  implies 

0  <  (1  -  a)aA  +  d2||c-(0)||  (U\\  +  lp||C-(0)||)  +  0(||dp||2).  (5.11) 

If  in  this  equation  we  use  the  bound  A  <  -0svh\\p\\2  an(I  we  ta^e  >nto  account  that  ||p|| 
is  bounded  away  from  zero  and  p  is  bounded,  by  taking  limits  along  the  subsequence  we 
obtain 

0  <  -(1  -  (j)0ivn(2, 

but  this  is  not  possible,  so  we  must  have  ||p*||  -♦  0  for  the  whole  sequence. 

The  other  possible  situation  corresponds  to  p*  growing  without  bound.  In  this  case  we 
have  that  for  all  iterations  where  the  value  of  the  penalty  parameter  is  increased 

P*,IM  <  and  Pt.llp*,!!2  <  AV 

The  proof  of  this  result  is  basically  that  of  Lemma  3.7.  From  these  bounds  it  is  possible  to 
show  that  we  must  also  have 

PfctlPfcll2  <  K 

for  all  k  (the  proof  is  similar  to  the  one  for  Lemma  4.6  in  [GMSW86b],  or  the  one  for 
Lemma  3.6.3  in  [Pr89]),  implying  p*  -»  0  and  the  convergence  of  the  algorithm.  | 

Near  the  solution  we  have  shown  a  minimizer  of  the  QP  subproblem  is  determined. 
Consequently,  the  issue  of  superlinear  convergence  is  not  relevant  to  whether  or  not  an 
incomplete  SQP  method  is  used  (once  convergence  has  been  established).  In  some  cases  the 
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algorithms  described  in  this  section  (for  specific  choices  of  multiplier  estimates)  are  simi¬ 
lar  to  known  algorithms  for  which  superlinear  convergence  is  not  always  attained  without 
some  modifications.  For  example,  it  has  been  demonstrated  that  the  algorithm  described 
in  [Han76]  may  not  take  a  unit  step  even  when  the  sequence  {xk  4-  Pk}  is  superlinearly 
convergent  (as  our  algorithm  does  when  using  our  original  merit  function). 

6.  Numerical  Results 

In  this  section  we  present  numerical  results  obtained  from  an  implementation  of  our  algo¬ 
rithm.  We  are  unaware  of  any  general  purpose  large-scale  SQP  routine  on  which  we  can 
test  our  modifications.  As  a  first  step  we  have  modified  the  code  NPSOL,  which  is  an 
implementation  of  the  algorithm  NPSQP.  We  have  called  the  modified  routine  INPSOL. 
Apart  from  the  definition  of  the  search  direction  all  other  aspects  of  INPSOL  are  identical 
to  those  of  NPSOL.  A  detailed  description  of  NPSOL  is  given  in  Gill  et  a I.  [GMSW86aj.  It 
should  be  noted  that  NPSOL  does  not  incorporate  linear  constraints  into  the  merit  func¬ 
tion.  An  initial  point  is  obtained  that  is  feasible  with  respect  to  the  linear  constraints  and 
thereafter  feasibility  is  retained  (by  incorporating  the  linear  constraints  in  the  QP  subprob¬ 
lem).  On  many  practical  problems  the  feasible  region  with  respect  to  the  linear  constraints 
is  compact.  On  such  problems  this  approach  ensures  assumption  A1  is  satisfied. 

The  purpose  of  the  testing  reported  is  to  demonstrate  that  the  efficiency  and  robustness 
of  the  modified  algorithm  are  comparable  to  those  of  NPSOL.  Naturally,  we  can  only  test  the 
hypothesis  on  the  domain  of  problems  NPSOL  is  designed  to  solve,  namely  problems  having 
a  small  number  of  variables  and  constraints,  although  on  these  problems  the  opportunities 
for  improvement  are  limited,  as  we  discuss  later.  What  this  implementation  really  tests  is 
whether  the  introduction  of  flexibility  in  the  determination  of  the  search  direction  has  a 
significant  cost. 

The  search  direction 

The  algorithm  described  in  Section  2  allows  for  considerable  flexibility  of  design.  We  de¬ 
scribe  here  the  specific  choices  made  in  our  implementation.  The  search  direction  pk  is 
computed  according  to  the  following  steps.  (The  subscript  k  corresponding  to  the  iteration 
number  is  dropped  from  now  on.) 

•  An  initial  feasible  point  p°  is  obtained  following  the  same  procedure  as  NPSOL.  No 
special  attempt  was  made  to  satisfy  condition  (2.7)  since  on  the  problems  tested  it 
was  always  satisfied  by  the  feasibility  phase  in  NPSOL. 

•  The  active-set  method  used  in  NPSOL  was  terminated  at  p,  the  first  stationary  point. 
At  p  the  corresponding  QP  multipliers  A  are  computed.  Define  A  as 

A.  =  A.IMI- 

•  Let  (M  denote  machine  precision.  If 

Vi  A«  > 


(6.1) 
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then  p  is  taken  as  the  search  direction. 

•  If  (6.1)  does  not  hold,  we  can  take  a  step  away  from  a  subset  of  the  active  constraints 
while  decreasing  the  value  of  the  QP  objective  function.  To  identify  the  set  of  active 
constraints  to  be  deleted,  define 


Pmin  — 


and  introduce  a  vector  e,  as 


f  ||®«l|  if  A«  — 

\  0  otherwise. 


(6.2) 


•  There  is  also  a  limit  of  50  on  the  maximum  number  of  constraints  to  be  deleted.  If 
(6.2)  is  satisfied  by  more  than  50  active  constraints,  only  the  ones  having  the  smallest 
multipliers  are  deleted.  For  most  problems  this  limit  has  no  effect,  since  the  total 
number  of  constraints  is  less  than  50. 

•  The  direction  away  from  the  selected  constraints  is  obtained  as  the  least-norm  solution 
of  the  system 

Ad  =■  e,; 

that  is,  we  define 

dY  =  (AYy'e,,  dz  =  0, 

to  obtain 

d  =  Ydy. 


•  If  ac  denotes  the  step  to  the  nearest  inactive  constraint,  and  am  is  defined  as  in  (2.9), 

{g  +  Hpfd 


— 


we  define  a  as  in  (2.9): 


dTHd  ’ 
a  =  rnin(ac,am,aM), 


where  aM  is  1010  for  this  case. 

•  We  obtain  the  search  direction  p  from  (2.10),  as 


-{% 


ad  >f  ||p||  <  0»ip\\P  +  a^ll> 
otherwise, 


where  f3aip  =  100;  with  this  value  the  step  ad  is  accepted  in  nearly  all  cases. 

•  Finally,  the  multiplier  estimate  used  to  define  the  linesearch  is  taken  to  be  A  if  P  =  A- 
Otherwise,  it  is  taken  to  be  the  least-squares  estimate  \L  obtained  from 

AAtXl  =  Ag . 
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Test  problems 

The  two  algorithms,  NPSOL  and  INPSOL,  have  been  compared  by  solving  a  collection  of 
114  problems  from  the  literature. 

The  problems  have  been  obtained  from  the  following  sources: 

•  Problem  1  is  the  example  problem  distributed  with  NPSOL;  its  description  can  be 
found  in  [GMSW86a].  Problems  3  and  4  are  slight  reformulations  of  the  same  problem, 
where  the  bounds  —  1  <  £3  <  1  have  been  replaced  by  the  constraint  x\  <  1.  Problem 
3  uses  the  starting  point 

(i  2  li  l  1  1  2  _i  _h 

V3’ 3’  10>  3' 3’ 3’  3’  3’  3'’ 

•  Descriptions  for  problems  6  and  12-15  can  be  found  in  [MS82].  The  version  of  problem 
6  considered  is  the  one  corresponding  to  a  value  T  —  10.  Problems  12  and  13  start 
from  point  (d)  for  Wright  No.  4  as  indicated  in  the  reference,  while  problems  14  and 
15  start  from  points  (a)  and  (b)  for  Wright  No.  9,  respectively. 

•  A  description  of  the  SQUARE  ROOT  problems  (17-20)  and  of  EXP6  (9)  can  be  found 
in  Fraley  [Fra88]. 

•  Problems  21-30  were  obtained  from  Boggs  and  Tolle  [BT84], 

•  All  problems  having  names  starting  with  “HS”  are  from  Hock  and  Schittkowski  [HS81]. 

•  Problems  85-95  can  be  found  in  Dembo  [Dem76]. 

All  the  above  problems  have  been  used  in  the  past  to  test  NPSOL.  It  should  be  noted 
that  the  problems  in  this  group  are  small;  the  average  number  of  variables  is  10,  and  the 
average  number  of  constraints  is  6.  Nevertheless,  many  of  these  problems  are  considered 
hard  to  solve.  Moreover,  for  some  of  these  problems  the  assumptions  made  to  establish  the 
convergence  results  fail  to  hold;  for  example,  in  some  cases  the  Jacobian  at  the  solution  is 
singular,  or  no  feasible  points  exist  for  some  QP  subproblems. 

The  algorithms  have  also  been  tested  on  another  group  of  problems. 

•  The  structural  optimization  problems  99-114  are  described  in  Ringertz  [Rin88].  The 
letters  “I”  and  “E”  in  the  problem  name  indicate  if  the  formulation  used  included 
explicitly  the  displacement  variables  (“E”)  or  eliminated  them  in  advance.  Also,  the 
following  number  (10,  25,  36  or  63)  denotes  the  number  of  bars  in  the  truss  considered. 
Finally,  whenever  a  number  is  included  at  the  end  of  the  name  (006,  040  or  060),  the 
initial  point  taken  has  been  modified  to  be  x}  =  6,  40  or  60  respectively. 

These  problems  have  been  introduced  due  to  the  atypical  behavior  of  quasi-Newton  SQP 
algorithms  on  them.  For  this  group,  the  ratio  of  QP  to  nonlinear  iterations  is  large  when 
compared  to  the  size  of  the  problem;  on  the  first  test  set  (problems  1-98)  the  average  ratio 
for  NPSOL  is  2  QP  iterations  per  nonlinear  iteration,  while  on  problems  99-114  the  average 
ratio  is  30. 
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The  normal  behavior  of  NPSOL  on  the  first  set  of  test  problems  is  to  require  a  relatively 
large  number  of  QP  iterations  in  the  first  few  nonlinear  iterations.  Typically,  the  number 
of  QP  iterations  declines  exponentially  until  near  the  solution,  when  only  one  iteration  is 
required.  The  STRUC  problems  depart  from  this  “standard”  behavior,  in  the  sense  that  the 
number  of  QP  iterations  declines  much  more  gradually.  (Although  only  one  QP  iteration 
is  required  in  the  end,  most  nonlinear  iterations  require  more.)  This  offers  the  possibility 
of  observing  the  reductions  that  can  be  achieved  by  using  the  early-termination  criterion, 
with  limited  distortion  from  the  asymptotic  behavior  of  NPSOL. 

Finally,  the  problems  in  this  second  group  are  larger  than  the  ones  presented  above; 
the  average  number  of  variables  is  now  55,  and  the  average  number  of  constraints  is  100. 
For  all  the  reasons  mentioned,  this  set  of  problems  provides  a  better  environment  in  which 
to  test  the  ability  of  the  proposed  early-termination  criterion  to  reduce  the  number  of  QP 
iterations. 

Computing  environment 

Version  4.02  of  NPSOL  was  used  in  these  comparisons.  For  this  test  set,  all  parameters 
used  in  the  code  have  been  fixed  at  their  default  values  (see  [GMSW86a]).  No  attempt  was 
made  to  improve  the  results  by  selecting  a  different  set  of  parameters.  It  would  be  difficult 
to  compare  the  relative  effort  to  adjust  input  parameters  for  the  two  algorithms.  The  runs 
were  performed  as  batch  jobs  on  a  DEC  VAXstation  II  with  5  Mb  main  memory.  The 
operating  system  was  VAX/VMS  version  4.5,  and  the  compiler  used  was  VAX  FORTRAN 
version  4.6  with  default  options. 

Results 

The  results  obtained  from  running  both  algorithms  on  the  test  set  are  presented  in  Table  2. 

The  parameters  chosen  to  characterize  the  relative  performance  of  both  algorithms  have 
been:  the  number  of  outer  (nonlinear)  iterations  for  each  problem;  the  number  of  calls  to 
the  routine  computing  the  values  of  the  objective  function,  the  constraint  functions  and 
their  derivatives  (function  evaluations);  the  total  number  of  inner  (QP)  iterations  for  the 
problem  (this  includes  the  number  of  iterations  necessary  to  compute  a  feasible  point);  and 
the  running  (CPU)  time  needed  to  solve  the  problem.  The  results  corresponding  to  both 
algorithms  are  given  as  a  single  entry  in  the  tables,  with  the  figures  separated  by  a  ”/” 
symbol,  in  the  form 

NPSOL  result/INPSOL  result. 

Given  that  many  of  the  problems  are  not  convex,  the  algorithms  may  converge  to  dif¬ 
ferent  solutions.  A  few  such  events  are  indicated  in  Table  2.  Another  possible  outcome  is 
failun1 — that  is,  the  algorithm  terminates  without  finding  a  solution,  because  the  iteration 
limit  has  been  exceeded,  because  no  significant  progress  can  be  made  at  the  current  point 
with  respect  to  the  merit  function,  or  because  the  objective  or  constraint  functions  need 
to  be  evaluated  at  a  point  for  which  they  are  not  defined  in  the  code.  Such  failures  are 
indicated  by  “ — ”. 
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For  the  set  of  114  problems,  NPSOL  was  able  to  find  a  solution  in  107  cases,  while 
INPSOL  was  able  to  solve  105  problems.  We  should  emphasize  that  only  the  default  value 
of  the  input  parameters  were  used.  Undoubtedly  adjustment  of  the  input  parameters  on  the 
problems  that  failed  would  have  led  to  more  successes.  The  figures  illustrate  the  reliability 
of  INPSOL. 

Table  1  presents  a  summary  of  the  results  for  the  four  quantities  monitored  in  Table  2. 
The  average  values  have  been  computed  as  the  geometric  means  for  the  ratios  of  the  values 
for  NPSOL  and  for  INPSOL;  that  is,  averages  larger  than  one  indicate  that  the  correspond¬ 
ing  value  for  NPSOL  is  larger  than  the  value  for  INPSOL.  Also,  the  averages  exclude  those 
problems  where  one  of  the  algorithms  failed.  Separate  entries  have  been  provided  for  prob¬ 
lems  1-98  (the  smaller  problems),  and  for  problems  99-114  (the  structural  optimization 
problems). 


Table  1 

Average  Behavior:  NPSOL  vs.  INPSOL 

Problems 


All 

1-98 

99-114 

Nonlinear  iterations 

.988 

.979 

1.044 

Function  evaluations 

.994 

.999 

.963 

QP  iterations 

1.190 

1.112 

1.884 

CPU  time 

1.043 

1.022 

1.200 

We  now  comment  briefly  on  the  implications  of  these  results. 

•  The  early-termination  rule  seems  to  behave  very  well  regarding  the  numbers  of  non¬ 
linear  iterations  and  function  evaluations;  even  if  we  are  now  using  a  search  direction 
of  “worse  quality”  than  in  NPSOL,  the  numbers  are  very  close  for  both  algorithms. 

•  The  number  of  QP  iterations  is  reduced  by  20%  for  the  complete  set.  When  judging 
this  figure  we  must  take  into  account  that  the  problems  are  small,  implying  that 
the  number  of  QP  iterations  required  per  nonlinear  iteration  is  also  small.  (In  fact, 
the  average  value  for  the  test  set  is  5.6  QP  iterations  per  nonlinear  iteration.)  The 
opportunity  for  improvement  is  correspondingly  limited.  Moreover,  both  codes  use  the 
active  set  at  the  solution  of  the  previous  QP  subproblem  as  a  prediction  for  the  correct 
active  set  in  the  current  subproblem,  resulting  in  a  small  number  of  QP  iterations  close 
to  the  solution.  As  a  result,  significant  savings  achieved  by  incomplete  solution  of  QP 
subproblems  in  the  early  iterations  are  masked  by  a  large  number  of  subproblems 
requiring  only  a  few  QP  iterations.  As  an  example,  for  problem  98  the  largest  number 


of  QP  iterations  needed  in  any  nonlinear  iteration  is  reduced  from  57  for  NPSOL  to 
15  for  INPSOL.  This  effect  is  much  less  clear  when  we  look  at  total  numbers  of  QP 
iterations  (244  for  NPSOL  vs.  170  for  INPSOL).  Recall  that  it  is  necessary  in  any 
implementation  to  limit  the  number  of  iterations  taken  to  solve  the  subproblem.  This 
large  reduction  in  the  maximum  number  of  iterations  is  encouraging.  Moreover,  it 
indicates  that  INPSOL  and  NPSOL  took  quite  different  paths  to  obtain  a  solution 
on  many  of  the  problems.  In  the  light  of  this  fact  the  similarity  of  performance  is 
quite  remarkable.  Finally,  the  early-termination  rule  still  requires  a  feasible  point, 
and  the  feasibility  phase  is  the  same  as  in  NPSOL.  When  this  phase  accounts  for 
most  of  the  total  number  of  iterations,  as  with  the  STRUC  problems,  the  possibility 
of  improvement  is  further  diminished. 

Nonetheless,  it  should  be  noted  that  for  problems  99-114  the  improvement  obtained 
is  significantly  greater  than  20%,  as  the  mean  ratio  is  now  1.88;  in  fact,  when  w®  look 
only  at  the  larger  problems,  the  relative  performance  of  INPSOL  improves  markedly. 
This  offers  the  promise  that  for  even  larger  problems  the  results  obtained  may  be 
substantially  better  than  the  values  shown  above. 

•  The  CPU  time  required  by  INPSOL  is  lower  than  the  time  for  NPSOL,  but  by  a 
factor  that  is  much  smaller  than  for  the  number  of  QP  iterations.  This  is  due  not 
only  to  the  fact  that  function  evaluations  can  be  expensive  when  compared  to  the 
effort  to  solve  each  QP  subproblem,  but  also  to  some  details  in  the  implementation 
that  have  been  chosen  to  affect  the  number  of  QP  iterations,  even  at  the  expense 
of  running  time.  For  example,  the  multiplier  estimate  used  for  the  linesearch  (the 
least-squares  multiplier)  is  expensive  to  compute  when  many  constraints  are  deleted 
in  the  last  step,  as  the  factorization  for  the  Jacobian  of  the  active  constraints  must 
be  updated.  There  are  still  options  to  be  explored  that  might  reduce  the  CPU  time 
for  the  modified  algorithm. 
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Table  2 
Numerical  Results 


No. 

Problem  name 

Nonlinear 

iterations 

Function 

evaluations 

QP 

iterations 

CPU 
time  (s) 

l 

NPSOL  SAMPLE  PROBLEM 

12/13 

16/18 

45/34 

3.69/3.61 

2 

SINGULAR 

15/15 

16/16 

4/4 

1.03/1.05 

3 

HEXAGON 

15/16 

21/23 

32/29 

4.41/4.41 

4 

HEXAGON  (ALT.  START) 

11/11 

16/14 

35/26 

3.56/3.26 

5 

LC7 

7/9 

9/11 

13/16 

.76/. 95 

6 

ALAN  MANNE’S  PROBLEM 

17/17 

18/18 

40/37 

21.13/21.92 

7 

ROSEN-SUZUKI 

8/8 

11/11 

9/9 

.81/.81 

8 

QP  PROBLEM 

8/l0 

9/11 

23/15 

1.10/1.04 

9 

EXP6 

33/53 

35/57 

38/57 

1.96/3.08 

10 

STEINKE2 

-*/5 

— /6 

/14 

— /.87 

11 

NORWAY 

4/6* 

5/7 

34/13 

1.23/.65 

12 

MHW4 

10/10 

18/15 

14/12 

1.31/1.25 

13 

MHW9 

30/19* 

56/28 

42/24 

3.71/2.31 

14 

MHW9  INEQUALITY  1 

28/23 

38/28 

59/40 

3.41/2.73 

15 

MHW9  INEQUALITY  2 

41/14* 

58/27 

80/24 

4.83/1.77 

16 

WOPLANT 

25/29 

29/33 

44/35 

6.85/7.17 

17 

SQUARE  ROOT  1 

— */— * 

-/- 

-/- 

-/- 

18 

SQUARE  ROOT  2 

23/23 

36/36 

0/0 

5.01/5.32 

19 

SQUARE  ROOT  3 

6/6 

9/9 

7/7 

.95/. 94 

20 

SQUARE  ROOT  4 

-*/-* 

-/- 

-/- 

/ — 

21 

BTl 

11/11 

19/19 

11/11 

.81/.83 

22 

BT2 

9/9 

14/14 

9/9 

.71/. 70 

23 

BT3 

2/2 

5/5 

2/2 

.19/. 19 

24 

BT4 

12/12 

18/18 

13/13 

.92/. 92 

25 

BT5-HS63 

6/6 

9/9 

8/8 

.58/. 58 

26 

BT6-HS77 

15/15 

21/21 

16/16 

1.52/1.54 

27 

BT7 

31/31 

56/56 

32/32 

3.36/3.43 

28 

BT8 

17/17 

19/19 

17/17 

1.25/1.44 

29 

BT9-HS39 

13/13 

16/16 

14/14 

.95/1.19 

30 

BT10 

8/8 

11/11 

0/0 

.48/. 52 

31 

BT11-HS79 

9/9 

12/12 

10/10 

1.05/1.06 

32 

BT12 

27/27 

57/57 

28/28 

3.04/3.04 

33 

BT13 

32/32 

44/44 

34/34 

2.61/2.62 

34 

POWELL  TRIANGLES 

23/15 

37/16 

36/23 

3.27/2.28 

35 

POWELL  BADLY  SCALED 

12/12 

15/15 

13/13 

.85/. 85 

36 

POWELL  WRIGGLE 

34/32 

69/55 

60/40 

2.77/2.39 

37 

POWELL-MARATOS 

6/6 

7/7 

6/6 

.44/. 44 

38 

HS72 

7/7 

8/8 

8/8 

.69/. 67 

39 

HS73  (CATTLE  FEED) 

4/4 

5/S 

4/4 

.38/. 36 

40 

HS107 

11/11 

18/18 

27/18 

2.77/2.56 

41 

MUKAI-POLAK 

10/10 

16/16 

13/13 

1.08/1.11 

42 

INFEASIBLE  SUBPROBLEM 

-*/-* 

-/- 

-/- 

-/- 

43 

HS26 

47/47 

64/64 

48/48 

3.39/3.41 

44 

HS32 

2/4 

3/5 

3/5 

.25/.38 

45 

HS46 

55/55 

58/58 

56/56 

5.26/4.98 

46 

HS51 

2/2 

5/5 

2/2 

.18/. 14 

47 

HS52 

2/2 

5/5 

2/2 

.19/. 16 

48 

HS53 

2/2 

5/5 

2/2 

.19/. 16 

49 

PENALTYl  A 

16/16 

18/19 

77/41 

20.01/16.49 

50 

PENALTY!  B 

6/7 

14/19 

67/32 

14.77/11.77 

51 

PENALTYl  C 

29/15 

85/40 

152/65 

24.35/11.65 

52 

HS13 

22/19 

23/20 

13/10 

1.29/1.22 

53 

HS64 

29/43 

39/62 

47/60 

2.34/3.33 

54 

HS65 

8/9 

10/11 

16/16 

.70/. 78 

55 

HS70 

36/—* 

39/- 

39/-  - 

3.33/— 

56 

HS71 

5/7 

6/9 

9/9 

.53/ .67 

57 

HS74 

10/26 

15/48 

14/28 

1.17/2.68 

*  Failed  to  solve  the  problem. 

*  Converged  to  a  different  minimizer. 
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Table  2  (cont.) 
Numerical  results 


No. 

Problem  name 

Nonlinear 

iterations 

Function 

evaluations 

QP 

iterations 

CPU 
time  (s) 

58 

HS75 

6/8 

10/11 

7/9 

.72/ .90 

59 

HS78 

10/10 

14/14 

11/11 

1.15/1.15 

60 

HS80 

8/8 

10/10 

8/8 

92/.92 

61 

HS81 

14/14 

20/20 

15/15 

1.57/1.60 

62 

HS84 

— */4 

— /5 

— /9 

— /.51 

63 

HS85 

17/14 

18/15 

33/20 

4.00/3.12 

64 

HS86  (COLVILLE  1) 

6/7 

8/8 

11/11 

.62/. 64 

65 

HS87  (COLVILLE  6) 

11/8 

18/9 

18/14 

1.63/1.23 

66 

HS93 

12/12 

15/15 

14/14 

1.36/1.38 

67 

HS95 

1/1 

2/2 

1/1 

.15/. 15 

68 

HS96 

1/1 

2/2 

1/1 

.17/. 15 

69 

HS97 

3/3 

6/6 

3/3 

.40/ .41 

70 

HS98 

3/3 

6/6 

8/8 

.43/. 44 

71 

HS99 

23/—* 

44/— 

74/- 

3.99/— 

72 

HS100 

14/14 

29/29 

18/18 

2.07/2.02 

73 

HS104 

18/18 

20/20 

23/23 

3.36/3.37 

74 

HS105 

43 /— * 

61/- 

97/— 

27.14/— 

75 

HS108  (HEXAGON) 

24/32 

45/49 

57/87 

6.78/9.36 

76 

HS109 

11/10 

13/11 

25/29 

3.23/3.26 

77 

HS110 

6/6 

9/9 

24/15 

.78/ .69 

78 

HS111 

41/49 

64/75 

44/52 

8.08/9.05 

79 

HS112  (CHEMICAL  EQ.) 

19/—* 

39/— 

54/- 

2.78/— 

80 

HS113 

14/16 

19/23 

38/36 

3.12/3.41 

81 

HS114 

18/16 

19/24 

36/33 

3.81/3.60 

82 

HS1 17  (COLVILLE  2) 

17/18 

21/27 

96/39 

6.75/5.34 

83 

HS118  (LC  PROBLEM) 

4/4 

6/6 

20/20 

1.35/1.40 

84 

HS119  (COLVILLE  7) 

12/17 

16/19 

41/47 

4.25/5.60 

85 

DEMB0  IB 

281/—* 

437/— 

296/— 

75.46/— 

86 

DEMBO  2-HS83 

4/4 

6/6 

4/4 

.54/. 54 

87 

DEMBO  3 

9/8 

ll/9 

37/20 

2.01/1.78 

88 

DEMBO  4A 

19/19 

23/23 

24/24 

3.53/3.31 

89 

DEMBO  4C 

13/13 

15/lj 

2U/ 23 

3.10/3.20 

90 

DEMBO  5-HS106 

17/18 

21/24 

30/31 

2.90/3.04 

91 

DEMBO  6-HS116 

36/43 

96/69 

144/248 

21.84/29.65 

92 

DEMBO  7 

19/12 

24/15 

126/68 

15.54/9.82 

93 

DEMBO  8A 

33/42 

85/118 

105/99 

7.52/9.17 

94 

DEMBO  8B 

29/29 

69/71 

88/73 

6.51/6.45 

95 

DEMBO  8C 

25/27 

60/68 

89/65 

6.19/6.06 

96 

OPF 

18/17 

19/18 

53/51 

468.12/456.10 

97 

GBD  EQUILIBRIUM  MOD. 

5/6 

6/7 

37/26 

6.22/6.10 

98 

WEAPON  ASSIGNMENT 

96/73 

98/76 

244/170 

120.78/114.93 

99 

STRUCIIOKON 

18/17 

34/30 

65/42 

13.67/11.73 

100 

STRUCE10KON 

26/29 

49/67 

87/84 

17.68/20.75 

101 

STRUCI10VAN 

23/19 

41/34 

54/51 

16.30/13.85 

102 

STRUCE10VAN 

— */24 

— /48 

— /91 

— /19.44 

103 

STRUCI25006 

42/37 

68/62 

147/85 

92.44/80.99 

104 

STRUCE25006 

20/28 

32/36 

178/95 

357.83/260.79 

105 

STRUCI25DAT 

1  l/l2 

19/21 

24/22 

24.75/27.11 

106 

STRUCE25DAT 

52/21 

106/37 

687/65 

647.13/191.44 

107 

STRUCI36DAT 

23/20 

38/34 

59/46 

120.79/108.02 

108 

STRUCE36DAT 

29/30 

53/62 

87/90 

971.16/1021.87 

109 

STRUCI63040 

117/112 

211/202 

6116/3091 

8182.13/7159.03 

no 

STRUCE63040 

375/—* 

794/— 

3545/— 

77286.64/— 

111 

STRUCI63060 

— */98 

— /244 

— /3899 

— /8281.02 

112 

STRUCE63060 

63/115 

150/316 

6675/3407 

25090.15/33228.42 

113 

STRUCI63DAT 

246/136 

354/412 

9043/2060 

12591.61/11424.54 

114  STRUCE63DAT 

*  Failed  to  solve  the  problem. 

52/72 

86/145 

8049/2858 

41793.84/22740.66 

*  Converged  to  a  different  minimizer. 
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